Merge pull request #5533 from FinnStutzenstein/externalAutoupdateService

External autoupdate service

.github/workflows/run-tests.yml

@@ -40,7 +40,7 @@ jobs:
         run: mypy openslides/ tests/
 
       - name: test using pytest
-        run: pytest --cov --cov-fail-under=75
+        run: pytest --cov --cov-fail-under=74
 
   install-client-dependencies:
     runs-on: ubuntu-latest

.gitignore

@@ -18,14 +18,12 @@
 .DS_Store
 Thumbs.db
 # Virtual Environment
-.virtualenv*/*
+.virtualenv*
 .venv
-server/.venv
 
 ## Compatibility
-# OS4-Submodules
+# OS4-Submodules and aux-directories
 /openslides-*/
-/haproxy/
 /docker/keys/
 /docs/
 # OS3+-Submodules
@@ -79,7 +77,8 @@ cypress.json
 
 ## Deployment
 # Docker build artifacts
-/docker/docker-compose.yml
+docker/docker-compose.yml
 *-version.txt
+*.pem
 # secrets
 docker/secrets/*.env

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "autoupdate"]
+	path = autoupdate
+	url = https://github.com/OpenSlides/openslides3-autoupdate-service.git

DEVELOPMENT.rst

@@ -2,187 +2,82 @@
  OpenSlides Development
 ========================
 
-This instruction helps you to setup a development environment for OpenSlides. A
+Check requirements
-simple dev setup will be configured without the need of the docker-compose
-setup. There are only the server running without a cache and a sqlite database
-and the client as an development server.
-
-
-1. Installation on GNU/Linux or Mac OS X
-----------------------------------------
-
-a. Check requirements
 '''''''''''''''''''''
 
-Make sure that you have installed `Python (>= 3.6) <https://www.python.org/>`_,
+- ``docker``
-`Node.js (>=10.x) <https://nodejs.org/>`_ and `Git <http://git-scm.com/>`_ on
+- ``docker-compose``
-your system. You also need build-essential packages and header files and a
+- ``git``
-static library for Python.
+- ``make``
 
-For Debian based systems (Ubuntu, etc) run::
+Note about migrating from previous OpenSlides3 development
-
+setups: You must set the ``OPENSLIDES_USER_DATA_DIR`` variable in
-    $ sudo apt-get install git nodejs npm build-essential python3-dev
+your ``server/personal_data/var/settings.py`` to  ``'/app/personal_data/var'``
 
 
-b. Get OpenSlides source code
+Get OpenSlides source code
 '''''''''''''''''''''''''''''
 
 Clone current master version from `OpenSlides GitHub repository
 <https://github.com/OpenSlides/OpenSlides/>`_::
 
-    $ git clone https://github.com/OpenSlides/OpenSlides.git
+    git clone https://github.com/OpenSlides/OpenSlides.git
-    $ cd OpenSlides
+    cd OpenSlides
+
+TODO: submodules.
+
+Start the development setup
+''''''''''''''''''''''''''''''
+
+    make run-dev
 
 
-c. Setup a virtual Python environment (optional)
+All you data (database, config, mediafiles) is stored in ``server/personal_data/var``.
-''''''''''''''''''''''''''''''''''''''''''''''''
 
-You can setup a virtual Python environment using the virtual environment
+Running the test cases
-(venv) package for Python to install OpenSlides as non-root user. This will
-allow for encapsulated dependencies. They will be installed in the virtual
-environment and not globally on your system.
-
-Setup and activate the virtual environment::
-
-    $ python3 -m venv .virtualenv
-    $ source .virtualenv/bin/activate
-
-You can exit the environment with::
-
-    $ deactivate
-
-d. Server
-'''''''''
-
-Go into the server's directory::
-
-    $ cd server/
-
-Install all required Python packages::
-
-    $ pip install --upgrade setuptools pip
-    $ pip install --requirement requirements.txt
-
-Create a settings file, run migrations and start the server::
-
-    $ python manage.py createsettings
-    $ python manage.py migrate
-    $ python manage.py runserver
-
-All you data (database, config, mediafiles) are stored in ``personal_data/var``.
-To get help on the command line options run::
-
-    $ python manage.py --help
-
-Later you might want to restart the server with one of the following commands.
-
-To run the OpenSlides server execute::
-
-    $ python manage.py runserver
-
-When debugging something email related change the email backend to console::
-
-    $ python manage.py runserver --debug-email
-
-The server is available under http://localhost:8000. Especially the rest interface
-might be important during development: http://localhost:8000/rest/ (The trailing
-slash is important!).
-
-e. Client
-'''''''''
-
-Go in the client's directory::
-
-    $ cd client/
-
-Install all dependencies and start the development server::
-
-    $ npm install
-    $ npm start
-
-After a while, the client is available under http://localhost:4200.
-
-
-2. Installation on Windows
---------------------------
-
-Follow the instructions above (Installation on GNU/Linux or Mac OS X) but care
-of the following variations.
-
-To get Python download and run the latest `Python 3.7 32-bit (x86) executable
-installer <https://www.python.org/downloads/windows/>`_. Note that the 32-bit
-installer is required even on a 64-bit Windows system. If you use the 64-bit
-installer, step d. of the instruction might fail unless you installed some
-packages manually.
-
-In some cases you have to install `MS Visual C++ 2015 build tools
-<https://www.microsoft.com/en-us/download/details.aspx?id=48159>`_ before you
-install the required python packages for OpenSlides (unfortunately Twisted
-needs it).
-
-To setup and activate the virtual environment in step c. use::
-
-    > .virtualenv\Scripts\activate.bat
-
-All other commands are the same as for GNU/Linux and Mac OS X.
-
-
-3. Running the test cases
 -------------------------
 
-a. Running server tests
+For all services in submodules check out the documentation there.
+
+Server tests andscripts
 '''''''''''''''''''''''
+You need to have python (>=3.8) and python-venv installed. Change your workdirectory to the server::
 
-To run some server tests see `.travis.yml
+    cd server
-<https://github.com/OpenSlides/OpenSlides/blob/master/.travis.yml>`_.
 
-b. Client tests and commands
+Setup an python virtual environment. If you have already done it, you can skip this step:
-''''''''''''''''''''''''''''
 
-Change to the client's directory to run every client related command. Run
+    python3 -m venv .venv
-client tests::
+    source .venv/bin/activate
+    pip install -U -r requirements.txt
 
-    $ npm test
+Make sure you are using the correct python version (e.g. try with explicit minor version: ``python3.8``). Activate it::
+
+    source .venv/bin/activate
+
+To deactivate it type ``deactivate``. Running all tests and linters:
+
+    black openslides/ tests/
+    flake8 openslides/ tests/
+    mypy openslides/ tests/
+    isort -rc openslides/ tests/
+    pytest tests/
+
+Client tests
+''''''''''''
+You need `node` and `npm` installed. Change to the client's directory. For the first time, install all dependencies::
+
+    cd client/
+    npm install
+
+Run client tests::
+
+    npm test
 
 Fix the code format and lint it with::
 
-    $ npm run prettify-write
+    npm run cleanup
-    $ npm run lint
 
 To extract translations run::
 
-    $ npm run extract
+    npm run extract
-
-When updating, adding or changing used packages from npm, please update the
-README.md using following command::
-
-    $ npm run licenses
-
-
-4. Notes for running OpenSlides in larger setups
-------------------------------------------------
-
-For productive setups refer to the docker-compose setup described in the main
-`README <https://github.com/OpenSlides/OpenSlides/blob/master/README.rst>`_.
-
-While develpment it might be handy to use a cache and another database.
-PostgreSQL is recommended and Redis necessary as a cache. Both can be set up in
-the ``settings.py``. Please consider reading the `OpenSlides configuration
-<https://github.com/OpenSlides/OpenSlides/blob/master/server/SETTINGS.rst>`_ page
-to find out about all configurations, especially when using OpenSlides for big
-assemblies.
-
-If you followed the instructions and installed the pip requirements form the
-``requirements.py`` all needed dependencies for another worker are installed.
-Instead of running ``python manage.py runserver`` you can use daphne or gunicorn
-(the latter is used in the prod setup)::
-
-    $ export DJANGO_SETTINGS_MODULE=settings
-    $ export PYTHONPATH=personal_data/var/
-    $ daphne -b 0.0.0.0 -p 8000 openslides.asgi:application
-
-The last line may be interchangeable with gunicorn and uvicorn as protocol
-server::
-
-    $ gunicorn -w 4 -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker openslides.asgi:application
-

Makefile

@@ -0,0 +1,17 @@
+build-dev:
+	make -C haproxy build-dev
+	git submodule foreach 'make build-dev'
+	docker-compose -f docker/docker-compose.dev.yml build
+
+run-dev: | build-dev
+	UID=$$(id -u $${USER}) GID=$$(id -g $${USER}) docker-compose -f docker/docker-compose.dev.yml up
+
+stop-dev:
+	docker-compose -f docker/docker-compose.dev.yml down
+
+reload-haproxy:
+	docker-compose -f docker/docker-compose.dev.yml kill -s HUP haproxy
+
+get-server-shell:
+	docker-compose -f docker/docker-compose.dev.yml run server bash
+

README.rst

@@ -25,6 +25,8 @@ First, you have to clone this repository::
     $ git clone https://github.com/OpenSlides/OpenSlides.git
     $ cd OpenSlides/docker/
 
+TODO: submodules.
+
 You need to build the Docker images for the client and server with this
 script::
 

autoupdate

@@ -0,0 +1 @@
+Subproject commit b187dd439bd7456e105901ca96cd0995862d4419

client/docker/Dockerfile

@@ -10,8 +10,8 @@ RUN npm install -g @angular/cli@^10
 RUN ng config -g cli.warnings.versionMismatch false
 
 USER openslides
-COPY package.json .
+COPY package.json package-lock.json ./
-COPY package-lock.json .
+RUN npm -v
 RUN npm ci
 COPY browserslist *.json ./
 COPY src ./src

client/docker/Dockerfile.dev

@@ -0,0 +1,11 @@
+FROM node:13
+
+WORKDIR /app
+
+COPY package.json .
+RUN npm install
+RUN npm run postinstall
+
+COPY . .
+
+CMD npm start

client/docker/nginx.conf

@@ -25,29 +25,6 @@ http {
         gzip_proxied expired no-cache no-store private auth;
         gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
 
-        location /apps {
-            proxy_pass http://server:8000;
-        }
-        location /media/ {
-            proxy_pass http://media:8000;
-        }
-        location /rest {
-            proxy_pass http://server:8000;
-        }
-        location /ws {
-            proxy_pass http://server:8000;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "Upgrade";
-        }
-        location /server-version.txt {
-            proxy_pass http://server:8000;
-        }
-
-        location = /basic_status {
-            stub_status;
-        }
-
         location / {
             try_files $uri $uri/ /index.html;
         }

client/ngsw-config.json

@@ -29,7 +29,7 @@
     "dataGroups": [
         {
             "name": "api",
-            "urls": ["/rest/*", "/apps/*"],
+            "urls": ["/rest/*", "/apps/*", "/system/*", "/stats"],
             "cacheConfig": {
                 "maxSize": 0,
                 "maxAge": "0u",

client/package.json

@@ -11,6 +11,7 @@
     "scripts": {
         "ng": "ng",
         "start": "ng serve --proxy-config proxy.conf.json --host=0.0.0.0",
+        "start-https": "ng serve --ssl --ssl-cert localhost.pem --ssl-key localhost-key.pem --proxy-config proxy.conf.json --host=0.0.0.0",
         "start-es5": "ng serve --proxy-config proxy.conf.json --host=0.0.0.0 --configuration es5",
         "build": "ng build --prod",
         "build-to-dir": "npm run build -- --output-path",

client/proxy.conf.json

@@ -15,5 +15,9 @@
         "target": "ws://localhost:8000",
         "secure": false,
         "ws": true
+    },
+    "/system/": {
+        "target": "https://localhost:8002",
+        "secure": false
     }
 }

client/src/app/app.component.ts

@@ -10,10 +10,9 @@ import { CountUsersService } from './core/ui-services/count-users.service';
 import { DataStoreUpgradeService } from './core/core-services/data-store-upgrade.service';
 import { LoadFontService } from './core/ui-services/load-font.service';
 import { LoginDataService } from './core/ui-services/login-data.service';
+import { OfflineService } from './core/core-services/offline.service';
 import { OperatorService } from './core/core-services/operator.service';
 import { OverlayService } from './core/ui-services/overlay.service';
-import { PingService } from './core/core-services/ping.service';
-import { PrioritizeService } from './core/core-services/prioritize.service';
 import { RoutingStateService } from './core/ui-services/routing-state.service';
 import { ServertimeService } from './core/core-services/servertime.service';
 import { ThemeService } from './core/ui-services/theme.service';
@@ -75,17 +74,16 @@ export class AppComponent {
         appRef: ApplicationRef,
         servertimeService: ServertimeService,
         router: Router,
+        offlineService: OfflineService,
         operator: OperatorService,
         loginDataService: LoginDataService,
-        constantsService: ConstantsService, // Needs to be started, so it can register itself to the WebsocketService
+        constantsService: ConstantsService,
         themeService: ThemeService,
         overlayService: OverlayService,
         countUsersService: CountUsersService, // Needed to register itself.
         configService: ConfigService,
         loadFontService: LoadFontService,
         dataStoreUpgradeService: DataStoreUpgradeService, // to start it.
-        prioritizeService: PrioritizeService,
-        pingService: PingService,
         routingState: RoutingStateService,
         votingBannerService: VotingBannerService // needed for initialisation
     ) {

client/src/app/core/core-services/autoupdate-throttle.service.ts

@@ -0,0 +1,164 @@
+import { EventEmitter, Injectable } from '@angular/core';
+
+import { Observable, Subject } from 'rxjs';
+
+import { AutoupdateFormat } from 'app/core/definitions/autoupdate-format';
+import { trailingThrottleTime } from 'app/core/rxjs/trailing-throttle-time';
+import { Identifiable } from 'app/shared/models/base/identifiable';
+import { ConstantsService } from './constants.service';
+import { HttpService } from './http.service';
+
+interface ThrottleSettings {
+    AUTOUPDATE_DELAY?: number;
+}
+
+@Injectable({
+    providedIn: 'root'
+})
+export class AutoupdateThrottleService {
+    private readonly _autoupdatesToInject = new Subject<AutoupdateFormat>();
+
+    public get autoupdatesToInject(): Observable<AutoupdateFormat> {
+        return this._autoupdatesToInject.asObservable();
+    }
+
+    private readonly receivedAutoupdate = new EventEmitter<void>();
+
+    private pendingAutoupdates = [];
+
+    private disabledUntil: number | null = null;
+
+    private delay = 0;
+
+    private maxSeenChangeId = 0;
+
+    private get isActive(): boolean {
+        return this.delay !== 0;
+    }
+
+    public constructor(private constantsService: ConstantsService, private httpService: HttpService) {
+        this.constantsService.get<ThrottleSettings>('Settings').subscribe(settings => {
+            // This is a one-shot. If the delay was set one time >0, it cannot be changed afterwards.
+            // A change is more complicated since you have to unsubscribe, clean pending autoupdates,
+            // subscribe again and make sure, that no autoupdate is missed.
+            if (this.delay === 0 && settings.AUTOUPDATE_DELAY) {
+                this.delay = 1000 * settings.AUTOUPDATE_DELAY;
+                console.log(`Configured autoupdate delay: ${this.delay}ms`);
+                this.receivedAutoupdate
+                    .pipe(trailingThrottleTime(this.delay))
+                    .subscribe(() => this.processPendingAutoupdates());
+            } else if (this.delay === 0) {
+                console.log('No autoupdate delay');
+            }
+        });
+
+        this.httpService.responseChangeIds.subscribe(changeId => this.disableUntil(changeId));
+    }
+
+    public newAutoupdate(autoupdate: AutoupdateFormat): void {
+        if (autoupdate.to_change_id > this.maxSeenChangeId) {
+            this.maxSeenChangeId = autoupdate.to_change_id;
+        }
+
+        if (!this.isActive) {
+            this._autoupdatesToInject.next(autoupdate);
+        } else if (this.disabledUntil !== null) {
+            this._autoupdatesToInject.next(autoupdate);
+            if (autoupdate.to_change_id >= this.disabledUntil) {
+                this.disabledUntil = null;
+                console.log('Throttling autoupdates again');
+            }
+        } else {
+            this.pendingAutoupdates.push(autoupdate);
+            this.receivedAutoupdate.emit();
+        }
+    }
+
+    public disableUntil(changeId: number): void {
+        // Wait for an autoupdate with to_id >= changeId.
+        console.log(this.delay);
+        if (!this.isActive) {
+            return;
+        }
+        this.processPendingAutoupdates();
+        // Checking with maxSeenChangeId is for the following race condition:
+        // If the autoupdate comes before the response, it must not be throttled.
+        // But flushing pending autoupdates is important since *if* the autoupdate
+        // was early, it is in the pending queue.
+        if (changeId <= this.maxSeenChangeId) {
+            return;
+        }
+        console.log('Disable autoupdate until change id', changeId);
+        this.disabledUntil = changeId;
+    }
+
+    private processPendingAutoupdates(): void {
+        const autoupdates = this.pendingAutoupdates;
+        if (autoupdates.length === 0) {
+            return;
+        }
+        this.pendingAutoupdates = [];
+
+        const autoupdate = this.mergeAutoupdates(autoupdates);
+        this._autoupdatesToInject.next(autoupdate);
+    }
+
+    private mergeAutoupdates(autoupdates: AutoupdateFormat[]): AutoupdateFormat {
+        const mergedAutoupdate: AutoupdateFormat = {
+            changed: {},
+            deleted: {},
+            from_change_id: autoupdates[0].from_change_id,
+            to_change_id: autoupdates[autoupdates.length - 1].to_change_id,
+            all_data: false
+        };
+
+        let lastToChangeId = null;
+        for (const au of autoupdates) {
+            if (lastToChangeId === null) {
+                lastToChangeId = au.to_change_id;
+            } else {
+                if (au.from_change_id !== lastToChangeId) {
+                    console.warn('!!!', autoupdates, au);
+                }
+                lastToChangeId = au.to_change_id;
+            }
+
+            this.applyAutoupdate(au, mergedAutoupdate);
+        }
+
+        return mergedAutoupdate;
+    }
+
+    private applyAutoupdate(from: AutoupdateFormat, into: AutoupdateFormat): void {
+        if (from.all_data) {
+            into.all_data = true;
+            into.changed = from.changed;
+            into.deleted = from.deleted;
+            return;
+        }
+
+        for (const collection of Object.keys(from.deleted)) {
+            for (const id of from.deleted[collection]) {
+                if (into.changed[collection]) {
+                    into.changed[collection] = into.changed[collection].filter(obj => (obj as Identifiable).id !== id);
+                }
+                if (!into.deleted[collection]) {
+                    into.deleted[collection] = [];
+                }
+                into.deleted[collection].push(id);
+            }
+        }
+
+        for (const collection of Object.keys(from.changed)) {
+            for (const obj of from.changed[collection]) {
+                if (into.deleted[collection]) {
+                    into.deleted[collection] = into.deleted[collection].filter(id => id !== (obj as Identifiable).id);
+                }
+                if (!into.changed[collection]) {
+                    into.changed[collection] = [];
+                }
+                into.changed[collection].push(obj);
+            }
+        }
+    }
+}

client/src/app/core/core-services/autoupdate.service.ts

@@ -1,57 +1,16 @@
 import { Injectable } from '@angular/core';
 
+import { AutoupdateFormat } from '../definitions/autoupdate-format';
+import { AutoupdateThrottleService } from './autoupdate-throttle.service';
 import { BaseModel } from '../../shared/models/base/base-model';
 import { CollectionStringMapperService } from './collection-string-mapper.service';
+import { CommunicationManagerService, OfflineError } from './communication-manager.service';
 import { DataStoreService, DataStoreUpdateManagerService } from './data-store.service';
+import { HttpService } from './http.service';
 import { Mutex } from '../promises/mutex';
-import { WebsocketService, WEBSOCKET_ERROR_CODES } from './websocket.service';
-
-export interface AutoupdateFormat {
-    /**
-     * All changed (and created) items as their full/restricted data grouped by their collection.
-     */
-    changed: {
-        [collectionString: string]: object[];
-    };
-
-    /**
-     * All deleted items (by id) grouped by their collection.
-     */
-    deleted: {
-        [collectionString: string]: number[];
-    };
-
-    /**
-     * The lower change id bond for this autoupdate
-     */
-    from_change_id: number;
-
-    /**
-     * The upper change id bound for this autoupdate
-     */
-    to_change_id: number;
-
-    /**
-     * Flag, if this autoupdate contains all data. If so, the DS needs to be resetted.
-     */
-    all_data: boolean;
-}
-
-export function isAutoupdateFormat(obj: any): obj is AutoupdateFormat {
-    const format = obj as AutoupdateFormat;
-    return (
-        obj &&
-        typeof obj === 'object' &&
-        format.changed !== undefined &&
-        format.deleted !== undefined &&
-        format.from_change_id !== undefined &&
-        format.to_change_id !== undefined &&
-        format.all_data !== undefined
-    );
-}
 
 /**
- * Handles the initial update and automatic updates using the {@link WebsocketService}
+ * Handles the initial update and automatic updates
  * Incoming objects, usually BaseModels, will be saved in the dataStore (`this.DS`)
  * This service usually creates all models
  */
@@ -61,32 +20,49 @@ export function isAutoupdateFormat(obj: any): obj is AutoupdateFormat {
 export class AutoupdateService {
     private mutex = new Mutex();
 
-    /**
+    private streamCloseFn: () => void | null = null;
-     * Constructor to create the AutoupdateService. Calls the constructor of the parent class.
+
-     * @param websocketService
+    private lastMessageContainedAllData = false;
-     * @param DS
+
-     * @param modelMapper
-     */
     public constructor(
-        private websocketService: WebsocketService,
         private DS: DataStoreService,
         private modelMapper: CollectionStringMapperService,
-        private DSUpdateManager: DataStoreUpdateManagerService
+        private DSUpdateManager: DataStoreUpdateManagerService,
+        private communicationManager: CommunicationManagerService,
+        private autoupdateThrottle: AutoupdateThrottleService
     ) {
-        this.websocketService.getOberservable<AutoupdateFormat>('autoupdate').subscribe(response => {
+        this.communicationManager.startCommunicationEvent.subscribe(() => this.startAutoupdate());
-            this.storeResponse(response);
-        });
 
-        // Check for too high change id-errors. If this happens, reset the DS and get fresh data.
+        this.autoupdateThrottle.autoupdatesToInject.subscribe(autoupdate => this.storeAutoupdate(autoupdate));
-        this.websocketService.errorResponseObservable.subscribe(error => {
+    }
-            if (error.code === WEBSOCKET_ERROR_CODES.CHANGE_ID_TOO_HIGH) {
+
-                this.doFullUpdate();
+    public async startAutoupdate(changeId?: number): Promise<void> {
+        this.stopAutoupdate();
+
+        try {
+            this.streamCloseFn = await this.communicationManager.subscribe<AutoupdateFormat>(
+                '/system/autoupdate',
+                autoupdate => {
+                    this.autoupdateThrottle.newAutoupdate(autoupdate);
+                },
+                () => ({ change_id: (changeId ? changeId : this.DS.maxChangeId).toString() })
+            );
+        } catch (e) {
+            if (!(e instanceof OfflineError)) {
+                console.error(e);
             }
-        });
+        }
+    }
+
+    public stopAutoupdate(): void {
+        if (this.streamCloseFn) {
+            this.streamCloseFn();
+            this.streamCloseFn = null;
+        }
     }
 
     /**
-     * Handle the answer of incoming data via {@link WebsocketService}.
+     * Handle the answer of incoming data, after it was throttled.
      *
      * Detects the Class of an incomming model, creates a new empty object and assigns
      * the data to it using the deserialize function. Also models that are flagged as deleted
@@ -94,8 +70,9 @@ export class AutoupdateService {
      *
      * Handles the change ids of all autoupdates.
      */
-    private async storeResponse(autoupdate: AutoupdateFormat): Promise<void> {
+    private async storeAutoupdate(autoupdate: AutoupdateFormat): Promise<void> {
         const unlock = await this.mutex.lock();
+        this.lastMessageContainedAllData = autoupdate.all_data;
         if (autoupdate.all_data) {
             await this.storeAllData(autoupdate);
         } else {
@@ -138,17 +115,10 @@ export class AutoupdateService {
         } else {
             // autoupdate fully in the future. we are missing something!
             console.log('Autoupdate in the future', maxChangeId, autoupdate.from_change_id, autoupdate.to_change_id);
-            this.requestChanges();
+            this.startAutoupdate(); // restarts it.
         }
     }
 
-    public async injectAutoupdateIgnoreChangeId(autoupdate: AutoupdateFormat): Promise<void> {
-        const unlock = await this.mutex.lock();
-        console.debug('inject autoupdate', autoupdate);
-        await this.injectAutupdateIntoDS(autoupdate, false);
-        unlock();
-    }
-
     private async injectAutupdateIntoDS(autoupdate: AutoupdateFormat, flush: boolean): Promise<void> {
         const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this.DS);
 
@@ -187,35 +157,21 @@ export class AutoupdateService {
         }
     }
 
-    /**
-     * Sends a WebSocket request to the Server with the maxChangeId of the DataStore.
-     * The server should return an autoupdate with all new data.
-     */
-    public requestChanges(): void {
-        console.log(`requesting changed objects with DS max change id ${this.DS.maxChangeId}`);
-        this.websocketService.send('getElements', { change_id: this.DS.maxChangeId });
-    }
-
     /**
      * Does a full update: Requests all data from the server and sets the DS to the fresh data.
      */
     public async doFullUpdate(): Promise<void> {
-        const oldChangeId = this.DS.maxChangeId;
+        if (this.lastMessageContainedAllData) {
-        const response = await this.websocketService.sendAndGetResponse<{}, AutoupdateFormat>('getElements', {});
+            console.log('full update requested. Skipping, last message already contained all data');
-
+        } else {
-        const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this.DS);
+            console.log('requesting full update.');
-        let allModels: BaseModel[] = [];
+            // The mutex is needed, so the DS is not cleared, if there is
-        for (const collection of Object.keys(response.changed)) {
+            // another autoupdate running.
-            if (this.modelMapper.isCollectionRegistered(collection)) {
+            const unlock = await this.mutex.lock();
-                allModels = allModels.concat(this.mapObjectsToBaseModels(collection, response.changed[collection]));
+            this.stopAutoupdate();
-            } else {
+            await this.DS.clear();
-                console.error(`Unregistered collection "${collection}". Ignore it.`);
+            this.startAutoupdate();
-            }
+            unlock();
         }
-
-        await this.DS.set(allModels, response.to_change_id);
-        this.DSUpdateManager.commit(updateSlot, response.to_change_id, true);
-
-        console.log(`Full update done from ${oldChangeId} to ${response.to_change_id}`);
     }
 }

client/src/app/core/core-services/communication-manager.service.ts

@@ -0,0 +1,186 @@
+import { HttpParams } from '@angular/common/http';
+import { EventEmitter, Injectable } from '@angular/core';
+
+import { Observable } from 'rxjs';
+
+import { HttpService } from './http.service';
+import { OfflineBroadcastService, OfflineReason } from './offline-broadcast.service';
+import { OperatorService } from './operator.service';
+import { SleepPromise } from '../promises/sleep';
+import {
+    CommunicationError,
+    ErrorType,
+    Stream,
+    StreamContainer,
+    StreamingCommunicationService,
+    verboseErrorType
+} from './streaming-communication.service';
+
+type HttpParamsGetter = () => HttpParams | { [param: string]: string | string[] };
+
+const MAX_STREAM_FAILURE_RETRIES = 3;
+
+export class OfflineError extends Error {
+    public constructor() {
+        super('');
+        this.name = 'OfflineError';
+    }
+}
+
+interface StreamConnectionWrapper {
+    id: number;
+    url: string;
+    messageHandler: (message: any) => void;
+    params: HttpParamsGetter;
+    stream?: Stream<any>;
+    hasErroredAmount: number;
+}
+
+@Injectable({
+    providedIn: 'root'
+})
+export class CommunicationManagerService {
+    private communicationAllowed = false;
+
+    private readonly _startCommunicationEvent = new EventEmitter<void>();
+
+    public get startCommunicationEvent(): Observable<void> {
+        return this._startCommunicationEvent.asObservable();
+    }
+
+    private readonly _stopCommunicationEvent = new EventEmitter<void>();
+
+    public get stopCommunicationEvent(): Observable<void> {
+        return this._stopCommunicationEvent.asObservable();
+    }
+
+    private streamContainers: { [id: number]: StreamContainer } = {};
+
+    public constructor(
+        private streamingCommunicationService: StreamingCommunicationService,
+        private offlineBroadcastService: OfflineBroadcastService,
+        private http: HttpService,
+        private operatorService: OperatorService
+    ) {
+        this.offlineBroadcastService.goOfflineObservable.subscribe(() => this.closeConnections());
+    }
+
+    public async subscribe<T>(
+        url: string,
+        messageHandler: (message: T) => void,
+        params?: HttpParamsGetter
+    ): Promise<() => void> {
+        if (!params) {
+            params = () => null;
+        }
+
+        const streamContainer = new StreamContainer(url, messageHandler, params);
+        return await this.connectWithWrapper(streamContainer);
+    }
+
+    public startCommunication(): void {
+        if (this.communicationAllowed) {
+            console.error('Illegal state! Do not emit this event multiple times');
+        } else {
+            this.communicationAllowed = true;
+            this._startCommunicationEvent.emit();
+        }
+    }
+
+    private async connectWithWrapper(streamContainer: StreamContainer): Promise<() => void> {
+        console.log('connect', streamContainer, streamContainer.stream);
+        const errorHandler = (type: ErrorType, error: CommunicationError, message: string) =>
+            this.handleError(streamContainer, type, error, message);
+        this.streamingCommunicationService.subscribe(streamContainer, errorHandler);
+        this.streamContainers[streamContainer.id] = streamContainer;
+        return () => this.close(streamContainer);
+    }
+
+    private async handleError(
+        streamContainer: StreamContainer,
+        type: ErrorType,
+        error: CommunicationError,
+        message: string
+    ): Promise<void> {
+        console.log('handle Error', streamContainer, streamContainer.stream, verboseErrorType(type), error, message);
+        streamContainer.stream.close();
+        streamContainer.stream = null;
+
+        streamContainer.hasErroredAmount++;
+        if (streamContainer.hasErroredAmount > MAX_STREAM_FAILURE_RETRIES) {
+            this.goOffline(streamContainer, OfflineReason.ConnectionLost);
+        } else if (type === ErrorType.Client && error.type === 'ErrorAuth') {
+            this.goOffline(streamContainer, OfflineReason.WhoAmIFailed);
+        } else {
+            // retry it after some time:
+            console.log(
+                `Retry no. ${streamContainer.hasErroredAmount} of ${MAX_STREAM_FAILURE_RETRIES} for ${streamContainer.url}`
+            );
+            try {
+                await this.delayAndCheckReconnection();
+                await this.connectWithWrapper(streamContainer);
+            } catch (e) {
+                // delayAndCheckReconnection can throw an OfflineError,
+                // which are just an 'abord mission' signal. Here, those errors can be ignored.
+            }
+        }
+    }
+
+    private async delayAndCheckReconnection(): Promise<void> {
+        const delay = Math.floor(Math.random() * 3000 + 2000);
+        console.log(`retry again in ${delay} ms`);
+
+        await SleepPromise(delay);
+
+        // do not continue, if we are offline!
+        if (this.offlineBroadcastService.isOffline()) {
+            console.log('we are offline?');
+            throw new OfflineError();
+        }
+
+        // do not continue, if we are offline!
+        if (!this.shouldRetryConnecting()) {
+            console.log('operator changed, do not rety');
+            throw new OfflineError(); // TODO: This error is not really good....
+        }
+    }
+
+    public closeConnections(): void {
+        for (const streamWrapper of Object.values(this.streamContainers)) {
+            if (streamWrapper.stream) {
+                streamWrapper.stream.close();
+            }
+        }
+        this.streamContainers = {};
+        this.communicationAllowed = false;
+        this._stopCommunicationEvent.emit();
+    }
+
+    private goOffline(streamContainer: StreamContainer, reason: OfflineReason): void {
+        delete this.streamContainers[streamContainer.id];
+        this.closeConnections(); // here we close the connections early.
+        this.offlineBroadcastService.goOffline(reason);
+    }
+
+    private close(streamConnectionWrapper: StreamConnectionWrapper): void {
+        if (this.streamContainers[streamConnectionWrapper.id]) {
+            this.streamContainers[streamConnectionWrapper.id].stream.close();
+            delete this.streamContainers[streamConnectionWrapper.id];
+        }
+    }
+
+    // Checks the operator: If we do not have a valid user,
+    // do not even try to connect again..
+    private shouldRetryConnecting(): boolean {
+        return this.operatorService.guestsEnabled || !!this.operatorService.user;
+    }
+
+    public async isCommunicationServiceOnline(): Promise<boolean> {
+        try {
+            const response = await this.http.get<{ healthy: boolean }>('/system/health');
+            return !!response.healthy;
+        } catch (e) {
+            return false;
+        }
+    }
+}

client/src/app/core/core-services/constants.service.ts

@@ -1,9 +1,11 @@
 import { Injectable } from '@angular/core';
 
+import { environment } from 'environments/environment';
 import { BehaviorSubject, Observable } from 'rxjs';
 import { filter } from 'rxjs/operators';
 
-import { WebsocketService } from './websocket.service';
+import { CommunicationManagerService } from './communication-manager.service';
+import { HttpService } from './http.service';
 
 /**
  * constants have a key associated with the data.
@@ -36,24 +38,15 @@ export class ConstantsService {
      */
     private subjects: { [key: string]: BehaviorSubject<any> } = {};
 
-    /**
+    public constructor(communicationManager: CommunicationManagerService, private http: HttpService) {
-     * @param websocketService
+        communicationManager.startCommunicationEvent.subscribe(async () => {
-     */
+            console.log('start communication');
-    public constructor(private websocketService: WebsocketService) {
+            this.constants = await this.http.get<Constants>(environment.urlPrefix + '/core/constants/');
-        // The hook for recieving constants.
+            console.log('constants:', this.constants);
-        websocketService.getOberservable<Constants>('constants').subscribe(constants => {
-            this.constants = constants;
             Object.keys(this.subjects).forEach(key => {
                 this.subjects[key].next(this.constants[key]);
             });
         });
-
-        // We can request constants, if the websocket connection opens.
-        // On retries, the `refresh()` method is called by the OpenSlidesService, so
-        // here we do not need to take care about this.
-        websocketService.noRetryConnectEvent.subscribe(() => {
-            this.refresh();
-        });
     }
 
     /**
@@ -66,14 +59,4 @@ export class ConstantsService {
         }
         return this.subjects[key].asObservable().pipe(filter(x => !!x));
     }
-
-    /**
-     * Refreshed the constants
-     */
-    public refresh(): Promise<void> {
-        if (!this.websocketService.isConnected) {
-            return;
-        }
-        this.websocketService.send('constants', {});
-    }
 }

client/src/app/core/core-services/data-store.service.ts

@@ -361,12 +361,12 @@ export class DataStoreService {
      *
      * @returns The max change id.
      */
-    public async initFromStorage(): Promise<number> {
+    public async initFromStorage(): Promise<void> {
         // This promise will be resolved with cached datastore.
         const store = await this.storageService.get<JsonStorage>(DataStoreService.cachePrefix + 'DS');
         if (!store) {
             await this.clear();
-            return this.maxChangeId;
+            return;
         }
 
         const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this);
@@ -395,7 +395,6 @@ export class DataStoreService {
             this.DSUpdateManager.dropUpdateSlot();
             await this.clear();
         }
-        return this.maxChangeId;
     }
 
     /**
@@ -670,6 +669,6 @@ export class DataStoreService {
     public print(): void {
         console.log('Max change id', this.maxChangeId);
         console.log(JSON.stringify(this.jsonStore));
-        console.log(this.modelStore);
+        console.log(JSON.parse(JSON.stringify(this.modelStore)));
     }
 }

client/src/app/core/core-services/http.service.ts

@@ -2,22 +2,14 @@ import { HttpClient, HttpErrorResponse, HttpHeaders } from '@angular/common/http
 import { Injectable } from '@angular/core';
 
 import { TranslateService } from '@ngx-translate/core';
+import { Subject } from 'rxjs';
 
-import { AutoupdateFormat, AutoupdateService, isAutoupdateFormat } from './autoupdate.service';
+import { AutoupdateFormat } from '../definitions/autoupdate-format';
+import { AutoupdateThrottleService } from './autoupdate-throttle.service';
+import { HTTPMethod } from '../definitions/http-methods';
 import { OpenSlidesStatusService } from './openslides-status.service';
 import { formatQueryParams, QueryParams } from '../definitions/query-params';
 
-/**
- * Enum for different HTTPMethods
- */
-export enum HTTPMethod {
-    GET = 'get',
-    POST = 'post',
-    PUT = 'put',
-    PATCH = 'patch',
-    DELETE = 'delete'
-}
-
 export interface ErrorDetailResponse {
     detail: string | string[];
     args?: string[];
@@ -33,12 +25,12 @@ function isErrorDetailResponse(obj: any): obj is ErrorDetailResponse {
 }
 
 interface AutoupdateResponse {
-    autoupdate: AutoupdateFormat;
+    change_id: number;
     data?: any;
 }
 
 function isAutoupdateReponse(obj: any): obj is AutoupdateResponse {
-    return obj && typeof obj === 'object' && isAutoupdateFormat((obj as AutoupdateResponse).autoupdate);
+    return obj && typeof obj === 'object' && typeof (obj as AutoupdateResponse).change_id === 'number';
 }
 
 /**
@@ -53,6 +45,8 @@ export class HttpService {
      */
     private defaultHeaders: HttpHeaders;
 
+    public readonly responseChangeIds = new Subject<number>();
+
     /**
      * Construct a HttpService
      *
@@ -65,8 +59,7 @@ export class HttpService {
     public constructor(
         private http: HttpClient,
         private translate: TranslateService,
-        private OSStatus: OpenSlidesStatusService,
+        private OSStatus: OpenSlidesStatusService
-        private autoupdateService: AutoupdateService
     ) {
         this.defaultHeaders = new HttpHeaders().set('Content-Type', 'application/json');
     }
@@ -205,8 +198,8 @@ export class HttpService {
 
     private processResponse<T>(responseData: T): T {
         if (isAutoupdateReponse(responseData)) {
-            this.autoupdateService.injectAutoupdateIgnoreChangeId(responseData.autoupdate);
+            this.responseChangeIds.next(responseData.change_id);
-            responseData = responseData.data;
+            return responseData.data;
         }
         return responseData;
     }

client/src/app/core/core-services/notify.service.ts

@@ -2,8 +2,9 @@ import { Injectable } from '@angular/core';
 
 import { Observable, Subject } from 'rxjs';
 
+import { CommunicationManagerService, OfflineError } from './communication-manager.service';
+import { HttpService } from './http.service';
 import { OperatorService } from './operator.service';
-import { WebsocketService } from './websocket.service';
 
 /**
  * Encapslates the name and content of every message regardless of being a request or response.
@@ -17,7 +18,12 @@ interface NotifyBase<T> {
     /**
      * The content to send.
      */
-    content: T;
+    message: T;
+}
+
+function isNotifyBase(obj: object): obj is NotifyResponse<any> {
+    const base = obj as NotifyBase<any>;
+    return !!obj && base.message !== undefined && base.name !== undefined;
 }
 
 /**
@@ -26,15 +32,18 @@ interface NotifyBase<T> {
  * channel names.
  */
 export interface NotifyRequest<T> extends NotifyBase<T> {
+    channel_id: string;
+    to_all?: boolean;
+
     /**
      * User ids (or `true` for all users) to send this message to.
      */
-    users?: number[] | boolean;
+    to_users?: number[];
 
     /**
      * An array of channels to send this message to.
      */
-    replyChannels?: string[];
+    to_channels?: string[];
 }
 
 /**
@@ -45,12 +54,12 @@ export interface NotifyResponse<T> extends NotifyBase<T> {
      * This is the channel name of the one, who sends this message. Can be use to directly
      * answer this message.
      */
-    senderChannelName: string;
+    sender_channel_id: string;
 
     /**
      * The user id of the user who sends this message. It is 0 for Anonymous.
      */
-    senderUserId: number;
+    sender_user_id: number;
 
     /**
      * This is validated here and is true, if the senderUserId matches the current operator's id.
@@ -59,6 +68,20 @@ export interface NotifyResponse<T> extends NotifyBase<T> {
     sendByThisUser: boolean;
 }
 
+function isNotifyResponse(obj: object): obj is NotifyResponse<any> {
+    const response = obj as NotifyResponse<any>;
+    // Note: we do not test for sendByThisUser, since it is set later in our code.
+    return isNotifyBase(obj) && response.sender_channel_id !== undefined && response.sender_user_id !== undefined;
+}
+
+interface ChannelIdResponse {
+    channel_id: string;
+}
+
+function isChannelIdResponse(obj: object): obj is ChannelIdResponse {
+    return !!obj && (obj as ChannelIdResponse).channel_id !== undefined;
+}
+
 /**
  * Handles all incoming and outgoing notify messages via {@link WebsocketService}.
  */
@@ -78,18 +101,41 @@ export class NotifyService {
         [name: string]: Subject<NotifyResponse<any>>;
     } = {};
 
-    /**
+    private channelId: string;
-     * Constructor to create the NotifyService. Registers itself to the WebsocketService.
+
-     * @param websocketService
+    public constructor(
-     */
+        private communicationManager: CommunicationManagerService,
-    public constructor(private websocketService: WebsocketService, private operator: OperatorService) {
+        private http: HttpService,
-        websocketService.getOberservable<NotifyResponse<any>>('notify').subscribe(notify => {
+        private operator: OperatorService
-            notify.sendByThisUser = notify.senderUserId === (this.operator.user ? this.operator.user.id : 0);
+    ) {
-            this.notifySubject.next(notify);
+        this.communicationManager.startCommunicationEvent.subscribe(() => this.startListening());
-            if (this.messageSubjects[notify.name]) {
+        this.communicationManager.stopCommunicationEvent.subscribe(() => (this.channelId = null));
-                this.messageSubjects[notify.name].next(notify);
+    }
+
+    private async startListening(): Promise<void> {
+        try {
+            await this.communicationManager.subscribe<NotifyResponse<any> | ChannelIdResponse>(
+                '/system/notify',
+                notify => {
+                    if (isChannelIdResponse(notify)) {
+                        this.channelId = notify.channel_id;
+                    } else if (isNotifyResponse(notify)) {
+                        notify.sendByThisUser =
+                            notify.sender_user_id === (this.operator.user ? this.operator.user.id : 0);
+                        this.notifySubject.next(notify);
+                        if (this.messageSubjects[notify.name]) {
+                            this.messageSubjects[notify.name].next(notify);
+                        }
+                    } else {
+                        console.error('Unknwon notify message', notify);
+                    }
+                }
+            );
+        } catch (e) {
+            if (!(e instanceof OfflineError)) {
+                console.log(e);
             }
-        });
+        }
     }
 
     /**
@@ -97,8 +143,8 @@ export class NotifyService {
      * @param name The name of the notify message
      * @param content The payload to send
      */
-    public sendToAllUsers<T>(name: string, content: T): void {
+    public async sendToAllUsers<T>(name: string, content: T): Promise<void> {
-        this.send(name, content);
+        await this.send(name, content, true);
     }
 
     /**
@@ -107,8 +153,11 @@ export class NotifyService {
      * @param content The payload to send.
      * @param users Multiple user ids.
      */
-    public sendToUsers<T>(name: string, content: T, ...users: number[]): void {
+    public async sendToUsers<T>(name: string, content: T, ...users: number[]): Promise<void> {
-        this.send(name, content, users);
+        if (users.length < 1) {
+            throw new Error('You have to provide at least one user');
+        }
+        await this.send(name, content, false, users);
     }
 
     /**
@@ -117,35 +166,48 @@ export class NotifyService {
      * @param content The payload to send.
      * @param channels Multiple channels to send this message to.
      */
-    public sendToChannels<T>(name: string, content: T, ...channels: string[]): void {
+    public async sendToChannels<T>(name: string, content: T, ...channels: string[]): Promise<void> {
         if (channels.length < 1) {
             throw new Error('You have to provide at least one channel');
         }
-        this.send(name, content, null, channels);
+        await this.send(name, content, false, null, channels);
     }
 
     /**
      * General send function for notify messages.
      * @param name The name of the notify message
-     * @param content The payload to send.
+     * @param message The payload to send.
      * @param users Either an array of IDs or `true` meaning of sending this message to all online users clients.
      * @param channels An array of channels to send this message to.
      */
-    public send<T>(name: string, content: T, users?: number[] | boolean, channels?: string[]): void {
+    private async send<T>(
+        name: string,
+        message: T,
+        toAll?: boolean,
+        users?: number[],
+        channels?: string[]
+    ): Promise<void> {
+        if (!this.channelId) {
+            throw new Error('No channel id!');
+        }
+
         const notify: NotifyRequest<T> = {
             name: name,
-            content: content
+            message: message,
+            channel_id: this.channelId
         };
-        if (typeof users === 'boolean' && users !== true) {
+        if (toAll === true) {
-            throw new Error('You just can give true as a boolean to send this message to all users.');
+            notify.to_all = true;
         }
-        if (users !== null) {
+        if (users) {
-            notify.users = users;
+            notify.to_users = users;
         }
-        if (channels !== null) {
+        if (channels) {
-            notify.replyChannels = channels;
+            notify.to_channels = channels;
         }
-        this.websocketService.send('notify', notify);
+
+        console.debug('send notify', notify);
+        await this.http.post<unknown>('/system/notify/send', notify);
     }
 
     /**

client/src/app/core/core-services/offline-broadcast.service.ts

@@ -0,0 +1,37 @@
+import { EventEmitter, Injectable } from '@angular/core';
+
+import { BehaviorSubject, Observable } from 'rxjs';
+
+export enum OfflineReason {
+    WhoAmIFailed,
+    ConnectionLost
+}
+
+@Injectable({
+    providedIn: 'root'
+})
+export class OfflineBroadcastService {
+    public readonly isOfflineSubject = new BehaviorSubject<boolean>(false);
+    public get isOfflineObservable(): Observable<boolean> {
+        return this.isOfflineSubject.asObservable();
+    }
+
+    private readonly _goOffline = new EventEmitter<OfflineReason>();
+    public get goOfflineObservable(): Observable<OfflineReason> {
+        return this._goOffline.asObservable();
+    }
+
+    public constructor() {}
+
+    public goOffline(reason: OfflineReason): void {
+        this._goOffline.emit(reason);
+    }
+
+    public isOffline(): boolean {
+        return this.isOfflineSubject.getValue();
+    }
+
+    public isOnline(): boolean {
+        return !this.isOffline();
+    }
+}

client/src/app/core/core-services/offline.service.ts

@@ -1,10 +1,9 @@
 import { Injectable } from '@angular/core';
 
-import { marker as _ } from '@biesbjerg/ngx-translate-extract-marker';
+import { CommunicationManagerService } from './communication-manager.service';
-import { TranslateService } from '@ngx-translate/core';
+import { OfflineBroadcastService, OfflineReason } from './offline-broadcast.service';
-import { BehaviorSubject, Observable } from 'rxjs';
+import { OpenSlidesService } from './openslides.service';
-
+import { OperatorService, WhoAmI } from './operator.service';
-import { BannerDefinition, BannerService } from '../ui-services/banner.service';
 
 /**
  * This service handles everything connected with being offline.
@@ -16,63 +15,111 @@ import { BannerDefinition, BannerService } from '../ui-services/banner.service';
     providedIn: 'root'
 })
 export class OfflineService {
-    /**
+    private reason: OfflineReason | null;
-     * BehaviorSubject to receive further status values.
-     */
-    private offline = new BehaviorSubject<boolean>(false);
-    private bannerDefinition: BannerDefinition = {
-        text: _('Offline mode'),
-        icon: 'cloud_off'
-    };
 
-    public constructor(private banner: BannerService, translate: TranslateService) {
+    public constructor(
-        translate.onLangChange.subscribe(() => {
+        private OpenSlides: OpenSlidesService,
-            this.bannerDefinition.text = translate.instant(this.bannerDefinition.text);
+        private offlineBroadcastService: OfflineBroadcastService,
-        });
+        private operatorService: OperatorService,
-    }
+        private communicationManager: CommunicationManagerService
-
+    ) {
-    /**
+        this.offlineBroadcastService.goOfflineObservable.subscribe((reason: OfflineReason) => this.goOffline(reason));
-     * Determines of you are either in Offline mode or not connected via websocket
-     *
-     * @returns whether the client is offline or not connected
-     */
-    public isOffline(): Observable<boolean> {
-        return this.offline;
-    }
-
-    /**
-     * Sets the offline flag. Restores the DataStoreService to the last known configuration.
-     */
-    public goOfflineBecauseFailedWhoAmI(): void {
-        if (!this.offline.getValue()) {
-            console.log('offline because whoami failed.');
-        }
-        this.goOffline();
-    }
-
-    /**
-     * Sets the offline flag, because there is no connection to the server.
-     */
-    public goOfflineBecauseConnectionLost(): void {
-        if (!this.offline.getValue()) {
-            console.log('offline because connection lost.');
-        }
-        this.goOffline();
     }
 
     /**
      * Helper function to set offline status
      */
-    private goOffline(): void {
+    public goOffline(reason: OfflineReason): void {
-        this.offline.next(true);
+        if (this.offlineBroadcastService.isOffline()) {
-        this.banner.addBanner(this.bannerDefinition);
+            return;
+        }
+        this.reason = reason;
+
+        if (reason === OfflineReason.ConnectionLost) {
+            console.log('offline because connection lost.');
+        } else if (reason === OfflineReason.WhoAmIFailed) {
+            console.log('offline because whoami failed.');
+        } else {
+            console.error('No such offline reason', reason);
+        }
+
+        this.offlineBroadcastService.isOfflineSubject.next(true);
+        this.checkStillOffline();
+    }
+
+    private checkStillOffline(): void {
+        const timeout = Math.floor(Math.random() * 3000 + 2000);
+        console.log(`Try to go online in ${timeout} ms`);
+
+        setTimeout(async () => {
+            let online: boolean;
+            let whoami: WhoAmI | null = null;
+
+            if (this.reason === OfflineReason.ConnectionLost) {
+                online = await this.communicationManager.isCommunicationServiceOnline();
+                console.log('is communication online? ', online);
+            } else if (this.reason === OfflineReason.WhoAmIFailed) {
+                const result = await this.operatorService.whoAmI();
+                online = result.online;
+                whoami = result.whoami;
+                console.log('is whoami reachable?', online);
+            }
+
+            if (online) {
+                await this.goOnline(whoami);
+                // TODO: check all other reasons -> e.g. if the
+                // connection was lost, the operator must be checked and the other way
+                // around the comminucation must be started!!
+
+                // stop trying.
+            } else {
+                // continue trying.
+                this.checkStillOffline();
+            }
+        }, timeout);
     }
 
     /**
      * Function to return to online-status.
+     *
+     * First, we have to check, if all other sources (except this.reason) are online, too.
+     * This results in definetly having a whoami response at this point.
+     * If this is the case, we need to setup everything again:
+     * 1) check the operator. If this allowes for an logged in state (or anonymous is OK), do
+     *    step 2, otherwise done.
+     * 2) enable communications.
      */
-    public goOnline(): void {
+    private async goOnline(whoami?: WhoAmI): Promise<void> {
-        this.offline.next(false);
+        console.log('go online!', this.reason, whoami);
-        this.banner.removeBanner(this.bannerDefinition);
+        if (this.reason === OfflineReason.ConnectionLost) {
+            // now we have to check whoami
+            const result = await this.operatorService.whoAmI();
+            if (!result.online) {
+                console.log('whoami down.');
+                this.reason = OfflineReason.WhoAmIFailed;
+                this.checkStillOffline();
+                return;
+            }
+            whoami = result.whoami;
+        } else if (this.reason === OfflineReason.WhoAmIFailed) {
+            const online = await this.communicationManager.isCommunicationServiceOnline();
+            if (!online) {
+                console.log('communication down.');
+                this.reason = OfflineReason.ConnectionLost;
+                this.checkStillOffline();
+                return;
+            }
+        }
+        console.log('we are online!');
+
+        // Ok, we are online now!
+        const isLoggedIn = await this.OpenSlides.checkWhoAmI(whoami);
+        console.log('logged in:', isLoggedIn);
+        if (isLoggedIn) {
+            this.communicationManager.startCommunication();
+        }
+        console.log('done');
+
+        this.offlineBroadcastService.isOfflineSubject.next(false);
     }
 }

client/src/app/core/core-services/openslides.service.ts

@@ -3,12 +3,11 @@ import { Router } from '@angular/router';
 
 import { BehaviorSubject } from 'rxjs';
 
-import { AutoupdateService } from './autoupdate.service';
+import { CommunicationManagerService } from './communication-manager.service';
-import { ConstantsService } from './constants.service';
 import { DataStoreService } from './data-store.service';
-import { OperatorService } from './operator.service';
+import { OfflineBroadcastService, OfflineReason } from './offline-broadcast.service';
+import { OperatorService, WhoAmI } from './operator.service';
 import { StorageService } from './storage.service';
-import { WebsocketService } from './websocket.service';
 
 /**
  * Handles the bootup/showdown of this application.
@@ -35,29 +34,19 @@ export class OpenSlidesService {
         return this.booted.value;
     }
 
-    /**
-     * Constructor to create the OpenSlidesService. Registers itself to the WebsocketService.
-     * @param storageService
-     * @param operator
-     * @param websocketService
-     * @param router
-     * @param autoupdateService
-     * @param DS
-     */
     public constructor(
         private storageService: StorageService,
         private operator: OperatorService,
-        private websocketService: WebsocketService,
         private router: Router,
-        private autoupdateService: AutoupdateService,
         private DS: DataStoreService,
-        private constantsService: ConstantsService
+        private communicationManager: CommunicationManagerService,
+        private offlineBroadcastService: OfflineBroadcastService
     ) {
         // Handler that gets called, if the websocket connection reconnects after a disconnection.
         // There might have changed something on the server, so we check the operator, if he changed.
-        websocketService.retryReconnectEvent.subscribe(() => {
+        /*websocketService.retryReconnectEvent.subscribe(() => {
             this.checkOperator();
-        });
+        });*/
 
         this.bootup();
     }
@@ -68,20 +57,24 @@ export class OpenSlidesService {
      */
     public async bootup(): Promise<void> {
         // start autoupdate if the user is logged in:
-        let response = await this.operator.whoAmIFromStorage();
+        let whoami = await this.operator.whoAmIFromStorage();
-        const needToCheckOperator = !!response;
+        const needToCheckOperator = !!whoami;
 
-        if (!response) {
+        if (!whoami) {
-            response = await this.operator.whoAmI();
+            const response = await this.operator.whoAmI();
+            if (!response.online) {
+                this.offlineBroadcastService.goOffline(OfflineReason.WhoAmIFailed);
+            }
+            whoami = response.whoami;
         }
 
-        if (!response.user && !response.guest_enabled) {
+        if (!whoami.user && !whoami.guest_enabled) {
             if (!location.pathname.includes('error')) {
                 this.redirectUrl = location.pathname;
             }
             this.redirectToLoginIfNotSubpage();
         } else {
-            await this.afterLoginBootup(response.user_id);
+            await this.afterLoginBootup(whoami.user_id);
         }
 
         if (needToCheckOperator) {
@@ -121,7 +114,7 @@ export class OpenSlidesService {
             await this.DS.clear();
             await this.storageService.set('lastUserLoggedIn', userId);
         }
-        await this.setupDataStoreAndWebSocket();
+        await this.setupDataStoreAndStartCommunication();
         // Now finally booted.
         this.booted.next(true);
     }
@@ -129,23 +122,16 @@ export class OpenSlidesService {
     /**
      * Init DS from cache and after this start the websocket service.
      */
-    private async setupDataStoreAndWebSocket(): Promise<void> {
+    private async setupDataStoreAndStartCommunication(): Promise<void> {
-        const changeId = await this.DS.initFromStorage();
+        await this.DS.initFromStorage();
-        // disconnect the WS connection, if there was one. This is needed
+        this.communicationManager.startCommunication();
-        // to update the connection parameters, namely the cookies. If the user
-        // is changed, the WS needs to reconnect, so the new connection holds the new
-        // user information.
-        if (this.websocketService.isConnected) {
-            await this.websocketService.close(); // Wait for the disconnect.
-        }
-        await this.websocketService.connect(changeId); // Request changes after changeId.
     }
 
     /**
-     * Shuts down OpenSlides. The websocket connection is closed and the operator is not set.
+     * Shuts down OpenSlides.
      */
     public async shutdown(): Promise<void> {
-        await this.websocketService.close();
+        this.communicationManager.closeConnections();
         this.booted.next(false);
     }
 
@@ -167,29 +153,37 @@ export class OpenSlidesService {
         await this.bootup();
     }
 
+    public async checkOperator(requestChanges: boolean = true): Promise<void> {
+        const response = await this.operator.whoAmI();
+        if (!response.online) {
+            this.offlineBroadcastService.goOffline(OfflineReason.WhoAmIFailed);
+        }
+        await this.checkWhoAmI(response.whoami, requestChanges);
+    }
+
     /**
      * Verify that the operator is the same as it was before. Should be alled on a reconnect.
+     *
+     * @returns true, if the user is still logged in
      */
-    private async checkOperator(requestChanges: boolean = true): Promise<void> {
+    public async checkWhoAmI(whoami: WhoAmI, requestChanges: boolean = true): Promise<boolean> {
-        const response = await this.operator.whoAmI();
+        let isLoggedIn = false;
         // User logged off.
-        if (!response.user && !response.guest_enabled) {
+        if (!whoami.user && !whoami.guest_enabled) {
-            this.websocketService.cancelReconnectenRetry();
             await this.shutdown();
             this.redirectToLoginIfNotSubpage();
         } else {
+            isLoggedIn = true;
             if (
-                (this.operator.user && this.operator.user.id !== response.user_id) ||
+                (this.operator.user && this.operator.user.id !== whoami.user_id) ||
-                (!this.operator.user && response.user_id)
+                (!this.operator.user && whoami.user_id)
             ) {
                 // user changed
                 await this.DS.clear();
                 await this.reboot();
-            } else if (requestChanges) {
-                // User is still the same, but check for missed autoupdates.
-                this.autoupdateService.requestChanges();
-                this.constantsService.refresh();
             }
         }
+
+        return isLoggedIn;
     }
 }

client/src/app/core/core-services/operator.service.ts

@@ -10,7 +10,6 @@ import { CollectionStringMapperService } from './collection-string-mapper.servic
 import { DataStoreService } from './data-store.service';
 import { Deferred } from '../promises/deferred';
 import { HttpService } from './http.service';
-import { OfflineService } from './offline.service';
 import { OnAfterAppsLoaded } from '../definitions/on-after-apps-loaded';
 import { OpenSlidesStatusService } from './openslides-status.service';
 import { StorageService } from './storage.service';
@@ -207,7 +206,6 @@ export class OperatorService implements OnAfterAppsLoaded {
     public constructor(
         private http: HttpService,
         private DS: DataStoreService,
-        private offlineService: OfflineService,
         private collectionStringMapper: CollectionStringMapperService,
         private storageService: StorageService,
         private OSStatus: OpenSlidesStatusService
@@ -306,18 +304,19 @@ export class OperatorService implements OnAfterAppsLoaded {
      *
      * @returns The response of the WhoAmI request.
      */
-    public async whoAmI(): Promise<WhoAmI> {
+    public async whoAmI(): Promise<{ whoami: WhoAmI; online: boolean }> {
+        let online = true;
         try {
             const response = await this.http.get(environment.urlPrefix + '/users/whoami/');
             if (isWhoAmI(response)) {
                 await this.updateCurrentWhoAmI(response);
             } else {
-                this.offlineService.goOfflineBecauseFailedWhoAmI();
+                online = false;
             }
         } catch (e) {
-            this.offlineService.goOfflineBecauseFailedWhoAmI();
+            online = false;
         }
-        return this.currentWhoAmI;
+        return { whoami: this.currentWhoAmI, online };
     }
 
     /**

client/src/app/core/core-services/ping.service.spec.ts

@@ -1,17 +0,0 @@
-import { inject, TestBed } from '@angular/core/testing';
-
-import { E2EImportsModule } from '../../../e2e-imports.module';
-import { PingService } from './ping.service';
-
-describe('PingService', () => {
-    beforeEach(() => {
-        TestBed.configureTestingModule({
-            imports: [E2EImportsModule],
-            providers: [PingService]
-        });
-    });
-
-    it('should be created', inject([PingService], (service: PingService) => {
-        expect(service).toBeTruthy();
-    }));
-});

client/src/app/core/core-services/ping.service.ts

@@ -1,86 +0,0 @@
-import { Injectable } from '@angular/core';
-
-import { ConstantsService } from './constants.service';
-import { Deferred } from '../promises/deferred';
-import { TimeoutPromise } from '../promises/timeout-promise';
-import { WebsocketService } from './websocket.service';
-
-interface OpenSlidesSettings {
-    PING_INTERVAL?: number;
-    PING_TIMEOUT?: number;
-}
-
-@Injectable({
-    providedIn: 'root'
-})
-export class PingService {
-    /**
-     * The interval.
-     */
-    private pingInterval: any;
-
-    private intervalTime = 30000;
-
-    private timeoutTime = 5000;
-
-    private lastLatency: number | null = null;
-
-    public constructor(private websocketService: WebsocketService, private constantsService: ConstantsService) {
-        this.setup();
-    }
-
-    private async setup(): Promise<void> {
-        const gotConstants = new Deferred();
-
-        this.constantsService.get<OpenSlidesSettings>('Settings').subscribe(settings => {
-            this.intervalTime = settings.PING_INTERVAL || 30000;
-            this.timeoutTime = settings.PING_TIMEOUT || 5000;
-            gotConstants.resolve();
-        });
-        await gotConstants;
-
-        // Connects the ping-pong mechanism to the opening and closing of the connection.
-        this.websocketService.closeEvent.subscribe(() => this.stopPing());
-        this.websocketService.generalConnectEvent.subscribe(() => this.startPing());
-        if (this.websocketService.isConnected) {
-            this.startPing();
-        }
-    }
-
-    /**
-     * Starts the ping-mechanism
-     */
-    private startPing(): void {
-        if (this.pingInterval) {
-            return;
-        }
-
-        this.pingInterval = setInterval(async () => {
-            const start = performance.now();
-            try {
-                await TimeoutPromise(
-                    this.websocketService.sendAndGetResponse('ping', this.lastLatency),
-                    this.timeoutTime
-                );
-                this.lastLatency = performance.now() - start;
-                if (this.lastLatency > 1000) {
-                    console.warn(`Ping took ${this.lastLatency / 1000} seconds.`);
-                }
-            } catch (e) {
-                console.warn(`The server didn't respond to ping within ${this.timeoutTime / 1000} seconds.`);
-                this.stopPing();
-                this.websocketService.simulateAbnormalClose();
-            }
-        }, this.intervalTime);
-    }
-
-    /**
-     * Clears the ping interval
-     */
-    private stopPing(): void {
-        if (this.pingInterval) {
-            clearInterval(this.pingInterval);
-            this.pingInterval = null;
-        }
-    }
-}

client/src/app/core/core-services/prioritize.service.spec.ts

@@ -1,17 +0,0 @@
-import { inject, TestBed } from '@angular/core/testing';
-
-import { E2EImportsModule } from '../../../e2e-imports.module';
-import { PrioritizeService } from './prioritize.service';
-
-describe('PrioritizeService', () => {
-    beforeEach(() => {
-        TestBed.configureTestingModule({
-            imports: [E2EImportsModule],
-            providers: [PrioritizeService]
-        });
-    });
-
-    it('should be created', inject([PrioritizeService], (service: PrioritizeService) => {
-        expect(service).toBeTruthy();
-    }));
-});

client/src/app/core/core-services/prioritize.service.ts

@@ -1,46 +0,0 @@
-import { Injectable } from '@angular/core';
-
-import { ConstantsService } from './constants.service';
-import { DataStoreService } from './data-store.service';
-import { OpenSlidesStatusService } from './openslides-status.service';
-import { OperatorService } from './operator.service';
-import { WebsocketService } from './websocket.service';
-
-interface OpenSlidesSettings {
-    PRIORITIZED_GROUP_IDS?: number[];
-}
-
-/**
- * Cares about prioritizing a client. Checks, if the operator is in one of
- * some prioritized groups. These group ids come from the server. If the prio-
- * ritization changes, the websocket connection gets reconnected.
- */
-@Injectable({
-    providedIn: 'root'
-})
-export class PrioritizeService {
-    private prioritizedGroupIds: number[] = [];
-
-    public constructor(
-        constantsService: ConstantsService,
-        private websocketService: WebsocketService,
-        private DS: DataStoreService,
-        private openSlidesStatusService: OpenSlidesStatusService,
-        private operator: OperatorService
-    ) {
-        constantsService.get<OpenSlidesSettings>('Settings').subscribe(settings => {
-            this.prioritizedGroupIds = settings.PRIORITIZED_GROUP_IDS || [];
-            this.checkPrioritization();
-        });
-        operator.getUserObservable().subscribe(() => this.checkPrioritization());
-    }
-
-    private checkPrioritization(): void {
-        const opPrioritized = this.operator.isInGroupIdsNonAdminCheck(...this.prioritizedGroupIds);
-        if (this.openSlidesStatusService.isPrioritizedClient !== opPrioritized) {
-            console.log('Alter prioritization:', opPrioritized);
-            this.openSlidesStatusService.isPrioritizedClient = opPrioritized;
-            this.websocketService.reconnect(this.DS.maxChangeId);
-        }
-    }
-}

client/src/app/core/core-services/projector-data.service.ts

@@ -3,8 +3,8 @@ import { Injectable } from '@angular/core';
 import { BehaviorSubject, Observable, Subject } from 'rxjs';
 import { auditTime } from 'rxjs/operators';
 
-import { WebsocketService } from 'app/core/core-services/websocket.service';
 import { Projector, ProjectorElement } from 'app/shared/models/core/projector';
+import { CommunicationManagerService, OfflineError } from './communication-manager.service';
 
 export interface SlideData<T = { error?: string }, P extends ProjectorElement = ProjectorElement> {
     data: T;
@@ -15,13 +15,13 @@ export interface SlideData<T = { error?: string }, P extends ProjectorElement =
 export type ProjectorData = SlideData[];
 
 interface AllProjectorData {
-    [id: number]: ProjectorData | { error: string };
+    [id: number]: ProjectorData;
 }
 
 /**
  * Received data from server.
  */
-interface ProjectorWebsocketMessage {
+interface ProjectorDataMessage {
     /**
      * The `change_id` of the current update.
      */
@@ -63,38 +63,63 @@ export class ProjectorDataService {
      */
     private currentChangeId = 0;
 
-    /**
+    private streamCloseFn: () => void | null = null;
-     * Constructor.
-     *
-     * @param websocketService
-     */
-    public constructor(private websocketService: WebsocketService) {
-        // Dispatch projector data.
-        this.websocketService.getOberservable('projector').subscribe((update: ProjectorWebsocketMessage) => {
-            if (this.currentChangeId > update.change_id) {
-                return;
-            }
-            Object.keys(update.data).forEach(_id => {
-                const id = parseInt(_id, 10);
-                if (this.currentProjectorData[id]) {
-                    this.currentProjectorData[id].next(update.data[id] as ProjectorData);
-                }
-            });
-            this.currentChangeId = update.change_id;
-        });
 
-        // The service need to re-register, if the websocket connection was lost.
+    public constructor(private communicationManager: CommunicationManagerService) {
-        this.websocketService.generalConnectEvent.subscribe(() => this.updateProjectorDataSubscription());
+        this.communicationManager.startCommunicationEvent.subscribe(() => this.updateProjectorDataSubscription());
 
         // With a bit of debounce, update the needed projectors.
         this.updateProjectorDataDebounceSubject.pipe(auditTime(10)).subscribe(() => {
             const allActiveProjectorIds = Object.keys(this.openProjectorInstances)
                 .map(id => parseInt(id, 10))
                 .filter(id => this.openProjectorInstances[id] > 0);
-            this.websocketService.send('listenToProjectors', { projector_ids: allActiveProjectorIds });
+            this.requestProjectors(allActiveProjectorIds);
         });
     }
 
+    public async requestProjectors(allActiveProjectorIds: number[]): Promise<void> {
+        this.cancelCurrentServerSubscription();
+
+        if (allActiveProjectorIds.length === 0) {
+            return;
+        }
+
+        try {
+            this.streamCloseFn = await this.communicationManager.subscribe<ProjectorDataMessage>(
+                '/system/projector',
+                message => {
+                    this.handleMesage(message);
+                },
+                () => ({ projector_ids: allActiveProjectorIds.join(',') })
+            );
+        } catch (e) {
+            if (!(e instanceof OfflineError)) {
+                console.log(e);
+            }
+        }
+    }
+
+    public cancelCurrentServerSubscription(): void {
+        if (this.streamCloseFn) {
+            this.streamCloseFn();
+            this.streamCloseFn = null;
+        }
+    }
+
+    private handleMesage(message: ProjectorDataMessage): void {
+        if (this.currentChangeId > message.change_id) {
+            console.log('Projector: Change id too low:', this.currentChangeId, message.change_id);
+            return;
+        }
+        Object.keys(message.data).forEach(_id => {
+            const id = parseInt(_id, 10);
+            if (this.currentProjectorData[id]) {
+                this.currentProjectorData[id].next(message.data[id] as ProjectorData);
+            }
+        });
+        this.currentChangeId = message.change_id;
+    }
+
     /**
      * Gets an observable for the projector data.
      *

client/src/app/core/core-services/projector.service.ts

@@ -40,12 +40,6 @@ export interface ProjectorTitle {
     providedIn: 'root'
 })
 export class ProjectorService {
-    /**
-     * Constructor.
-     *
-     * @param DS
-     * @param dataSend
-     */
     public constructor(
         private DS: DataStoreService,
         private http: HttpService,

client/src/app/core/core-services/streaming-communication.service.ts

@@ -0,0 +1,281 @@
+import {
+    HttpClient,
+    HttpDownloadProgressEvent,
+    HttpEvent,
+    HttpHeaderResponse,
+    HttpHeaders,
+    HttpParams
+} from '@angular/common/http';
+import { Injectable } from '@angular/core';
+
+import { Observable, Subscription } from 'rxjs';
+
+import { HTTPMethod } from '../definitions/http-methods';
+
+const HEADER_EVENT_TYPE = 2;
+const PROGRESS_EVENT_TYPE = 3;
+const FINISH_EVENT_TYPE = 4;
+
+export type Params = HttpParams | { [param: string]: string | string[] };
+export enum ErrorType {
+    Client,
+    Server, // or network errors, they are the same.
+    Unknown
+}
+export function verboseErrorType(type: ErrorType): string {
+    switch (type) {
+        case ErrorType.Client:
+            return 'Client';
+        case ErrorType.Server:
+            return 'Server';
+        case ErrorType.Unknown:
+            return 'Unknown';
+    }
+}
+export interface CommunicationError {
+    type: string;
+    msg: string;
+}
+export function isCommunicationError(obj: any): obj is CommunicationError {
+    const _obj = obj as CommunicationError;
+    return typeof obj === 'object' && typeof _obj.msg === 'string' && typeof _obj.type === 'string';
+}
+interface CommunicationErrorWrapper {
+    error: CommunicationError;
+}
+export function isCommunicationErrorWrapper(obj: any): obj is CommunicationErrorWrapper {
+    return typeof obj === 'object' && isCommunicationError(obj.error);
+}
+export type ErrorHandler = (type: ErrorType, error: CommunicationError, message: string) => void;
+
+export class StreamConnectionError extends Error {
+    public constructor(public code: number, message: string) {
+        super(message);
+        this.name = 'StreamConnectionError';
+    }
+}
+
+export class StreamContainer {
+    public readonly id = Math.floor(Math.random() * (900000 - 1) + 100000); // [100000, 999999]
+
+    public hasErroredAmount = 0;
+
+    public stream?: Stream<any>;
+
+    public constructor(
+        public url: string,
+        public messageHandler: (message: any) => void,
+        public params: () => Params
+    ) {}
+}
+
+export class Stream<T> {
+    private subscription: Subscription = null;
+
+    private hasError = false;
+    private reportedError = false;
+
+    private _statuscode: number;
+    public get statuscode(): number {
+        return this._statuscode;
+    }
+
+    private _errorContent: CommunicationError;
+    public get errorContent(): CommunicationError {
+        return this._errorContent;
+    }
+
+    /**
+     * This is the index where we checked, if there is a \n in the read buffer (event.partialText)
+     * This position is always >= contentStartIndex and is > contentStartIndex, if the message
+     * was too big to fit into one buffer. So we have just a partial message.
+     *
+     * The difference between this index and contentStartIndex is that this index remembers the position
+     * we checked for a \n which lay in the middle of the next JOSN-packet.
+     */
+    private checkedUntilIndex = 0;
+
+    /**
+     * This index holds always the position of the current JOSN-packet, that we are receiving.
+     */
+    private contentStartIndex = 0;
+
+    private closed = false;
+
+    public constructor(
+        observable: Observable<HttpEvent<string>>,
+        private messageHandler: (message: T) => void,
+        private errorHandler: ErrorHandler
+    ) {
+        this.subscription = observable.subscribe(
+            (event: HttpEvent<string>) => {
+                if (this.closed) {
+                    return;
+                }
+                if (event.type === HEADER_EVENT_TYPE) {
+                    const headerResponse = event as HttpHeaderResponse;
+                    this._statuscode = headerResponse.status;
+                    if (headerResponse.status >= 400) {
+                        this.hasError = true;
+                    }
+                } else if ((<HttpEvent<string>>event).type === PROGRESS_EVENT_TYPE) {
+                    this.handleMessage(event as HttpDownloadProgressEvent);
+                } else if ((<HttpEvent<string>>event).type === FINISH_EVENT_TYPE) {
+                    this.errorHandler(ErrorType.Server, null, 'The stream was closed');
+                }
+            },
+            error => {
+                this.errorHandler(ErrorType.Server, error, 'Network error');
+            },
+            () => {
+                console.log('The stream was completed');
+            }
+        );
+    }
+
+    private handleMessage(event: HttpDownloadProgressEvent): void {
+        if (this.hasError) {
+            if (!this.reportedError) {
+                this.reportedError = true;
+                // try to get the `error` key from object
+                this._errorContent = this.tryParseError(event.partialText);
+                this.errorHandler(this.getErrorTypeFromStatusCode(), this._errorContent, 'Reported error by server');
+            }
+            return;
+        }
+        // Maybe we get multiple messages, so continue, until the complete buffer is checked.
+        while (this.checkedUntilIndex < event.loaded) {
+            // check if there is a \n somewhere in [checkedUntilIndex, ...]
+            const LF_index = event.partialText.indexOf('\n', this.checkedUntilIndex);
+
+            if (LF_index >= 0) {
+                // take string in [contentStartIndex, LF_index-1]. This must be valid JSON.
+                // In substring, the last character is exlusive.
+                const content = event.partialText.substring(this.contentStartIndex, LF_index);
+
+                // move pointer: next JSON starts at LF_index + 1
+                this.checkedUntilIndex = LF_index + 1;
+                this.contentStartIndex = LF_index + 1;
+
+                const parsedContent = this.tryParseJson(content);
+                if (isCommunicationError(parsedContent)) {
+                    if (this.hasError && this.reportedError) {
+                        return;
+                    }
+                    this.hasError = true;
+                    this._errorContent = parsedContent;
+                    // Do not trigger the error handler, if the connection-retry-routine is still handling this issue
+                    if (!this.reportedError) {
+                        this.reportedError = true;
+                        console.error(this._errorContent);
+                        this.errorHandler(
+                            this.getErrorTypeFromStatusCode(),
+                            this._errorContent,
+                            'Reported error by server'
+                        );
+                    }
+                    return;
+                } else {
+                    // check, if we didn't get a keep alive
+                    if (Object.keys(parsedContent).length > 0) {
+                        console.debug('received', parsedContent);
+                        this.messageHandler(parsedContent);
+                    }
+                }
+            } else {
+                this.checkedUntilIndex = event.loaded;
+            }
+        }
+    }
+
+    private getErrorTypeFromStatusCode(): ErrorType {
+        if (!this.statuscode) {
+            return ErrorType.Unknown;
+        }
+        if (this.statuscode >= 400 && this.statuscode < 500) {
+            return ErrorType.Client;
+        }
+        if (this.statuscode >= 500) {
+            return ErrorType.Server;
+        }
+        return ErrorType.Unknown;
+    }
+
+    private tryParseJson(json: string): T | CommunicationError {
+        try {
+            return JSON.parse(json) as T;
+        } catch (e) {
+            return this.tryParseError(json);
+        }
+    }
+
+    /**
+     * This one is a bit tricky. Error can be:
+     * - string with HTML, e.g. provided by proxies if the service is unreachable
+     * - string with json of form {"error": {"type": ..., "msg": ...}}
+     */
+    private tryParseError(error: any): CommunicationError {
+        if (typeof error === 'string') {
+            try {
+                error = JSON.parse(error);
+            } catch (e) {
+                return { type: 'Unknown Error', msg: error };
+            }
+        }
+
+        if (isCommunicationErrorWrapper(error)) {
+            return error.error;
+        } else if (isCommunicationError(error)) {
+            return error;
+        }
+
+        // we have something else.... ??
+        console.error('Unknown error', error);
+        throw new Error('Unknown error: ' + error.toString());
+    }
+
+    public close(): void {
+        this.subscription.unsubscribe();
+        this.subscription = null;
+        this.closed = true;
+    }
+}
+
+@Injectable({
+    providedIn: 'root'
+})
+export class StreamingCommunicationService {
+    public constructor(private http: HttpClient) {}
+
+    public subscribe<T>(streamContainer: StreamContainer, errorHandler: ErrorHandler): void {
+        const options: {
+            body?: any;
+            headers?: HttpHeaders | { [header: string]: string | string[] };
+            observe: 'events';
+            params?: HttpParams | { [param: string]: string | string[] };
+            reportProgress?: boolean;
+            responseType: 'text';
+            withCredentials?: boolean;
+        } = {
+            headers: {
+                'Content-Type': 'application/json',
+                'ngsw-bypass': 'yes'
+            },
+            responseType: 'text',
+            observe: 'events',
+            reportProgress: true
+        };
+        const params = streamContainer.params();
+        if (params) {
+            options.params = params;
+        }
+        const observable = this.http.request(HTTPMethod.GET, streamContainer.url, options);
+
+        if (streamContainer.stream) {
+            console.error('Illegal state!');
+        }
+
+        const stream = new Stream<T>(observable, streamContainer.messageHandler, errorHandler);
+        streamContainer.stream = stream;
+    }
+}

client/src/app/core/core-services/time-travel.service.ts

@@ -9,7 +9,6 @@ import { DataStoreService, DataStoreUpdateManagerService } from './data-store.se
 import { HttpService } from './http.service';
 import { OpenSlidesStatusService } from './openslides-status.service';
 import { OpenSlidesService } from './openslides.service';
-import { WebsocketService } from './websocket.service';
 
 interface HistoryData {
     [collection: string]: BaseModel[];
@@ -31,7 +30,6 @@ export class TimeTravelService {
      * Constructs the time travel service
      *
      * @param httpService To fetch the history data
-     * @param webSocketService to disable websocket connection
      * @param modelMapperService to cast history objects into models
      * @param DS to overwrite the dataStore
      * @param OSStatus Sets the history status
@@ -39,7 +37,6 @@ export class TimeTravelService {
      */
     public constructor(
         private httpService: HttpService,
-        private webSocketService: WebsocketService,
         private modelMapperService: CollectionStringMapperService,
         private DS: DataStoreService,
         private OSStatus: OpenSlidesStatusService,
@@ -100,7 +97,8 @@ export class TimeTravelService {
      * Clears the DataStore and stops the WebSocket connection
      */
     private async stopTime(history: History): Promise<void> {
-        await this.webSocketService.close();
+        // await this.webSocketService.close();
+        // TODO
         await this.DS.set(); // Same as clear, but not persistent.
         this.OSStatus.enterHistoryMode(history);
     }

client/src/app/core/core-services/websocket.service.spec.ts

@@ -1,17 +0,0 @@
-import { inject, TestBed } from '@angular/core/testing';
-
-import { E2EImportsModule } from '../../../e2e-imports.module';
-import { WebsocketService } from './websocket.service';
-
-describe('WebsocketService', () => {
-    beforeEach(() => {
-        TestBed.configureTestingModule({
-            imports: [E2EImportsModule],
-            providers: [WebsocketService]
-        });
-    });
-
-    it('should be created', inject([WebsocketService], (service: WebsocketService) => {
-        expect(service).toBeTruthy();
-    }));
-});

client/src/app/core/core-services/websocket.service.ts

@@ -1,551 +0,0 @@
-import { EventEmitter, Injectable, NgZone } from '@angular/core';
-import { MatSnackBarRef, SimpleSnackBar } from '@angular/material/snack-bar';
-import { Router } from '@angular/router';
-
-import { compress, decompress } from 'lz4js';
-import { Observable, Subject } from 'rxjs';
-import { take } from 'rxjs/operators';
-
-import { OfflineService } from './offline.service';
-import { OpenSlidesStatusService } from './openslides-status.service';
-import { formatQueryParams, QueryParams } from '../definitions/query-params';
-
-/**
- * The generic message format in which messages are send and recieved by the server.
- */
-interface BaseWebsocketMessage {
-    type: string;
-    content: any;
-}
-
-/**
- * Outgoing messages must have an id.
- */
-interface OutgoingWebsocketMessage extends BaseWebsocketMessage {
-    id: string;
-}
-
-/**
- * Incomming messages may have an `in_response`, if they are an answer to a previously
- * submitted request.
- */
-interface IncommingWebsocketMessage extends BaseWebsocketMessage {
-    in_response?: string;
-}
-
-/**
- * The format of a messages content, if the message type is "error"
- */
-interface WebsocketErrorContent {
-    code: number;
-    message: string;
-}
-
-function isWebsocketErrorContent(obj: any): obj is WebsocketErrorContent {
-    return !!obj && obj.code !== undefined && obj.message !== undefined;
-}
-
-/**
- * All (custom) error codes that are used to pass error information
- * from the server to the client
- */
-export const WEBSOCKET_ERROR_CODES = {
-    NOT_AUTHORIZED: 100,
-    CHANGE_ID_TOO_HIGH: 101,
-    WRONG_FORMAT: 102
-};
-
-/**
- * Service that handles WebSocket connections. Other services can register themselfs
- * with {@method getOberservable} for a specific type of messages. The content will be published.
- */
-@Injectable({
-    providedIn: 'root'
-})
-export class WebsocketService {
-    /**
-     * The reference to the snackbar entry that is shown, if the connection is lost.
-     */
-    private connectionErrorNotice: MatSnackBarRef<SimpleSnackBar>;
-
-    /**
-     * Subjects that will be called, if a reconnect after a retry (e.g. with a previous
-     * connection loss) was successful.
-     */
-    private readonly _retryReconnectEvent: EventEmitter<void> = new EventEmitter<void>();
-
-    /**
-     * Getter for the retry reconnect event.
-     */
-    public get retryReconnectEvent(): EventEmitter<void> {
-        return this._retryReconnectEvent;
-    }
-
-    /**
-     * Subjects that will be called, if connect took place, but not a retry reconnect.
-     * THis is the complement from the generalConnectEvent to the retryReconnectEvent.
-     */
-    private readonly _noRetryConnectEvent: EventEmitter<void> = new EventEmitter<void>();
-
-    /**
-     * Getter for the no-retry connect event.
-     */
-    public get noRetryConnectEvent(): EventEmitter<void> {
-        return this._noRetryConnectEvent;
-    }
-
-    /**
-     * Listeners will be nofitied, if the wesocket connection is establiched.
-     */
-    private readonly _generalConnectEvent: EventEmitter<void> = new EventEmitter<void>();
-
-    /**
-     * Getter for the connect event.
-     */
-    public get generalConnectEvent(): EventEmitter<void> {
-        return this._generalConnectEvent;
-    }
-
-    /**
-     * Listeners will be nofitied, if the wesocket connection is closed.
-     */
-    private readonly _closeEvent: EventEmitter<void> = new EventEmitter<void>();
-
-    /**
-     * Getter for the close event.
-     */
-    public get closeEvent(): EventEmitter<void> {
-        return this._closeEvent;
-    }
-
-    /**
-     * The subject for all websocket *message* errors (no connection errors).
-     */
-    private readonly _errorResponseSubject = new Subject<WebsocketErrorContent>();
-
-    /**
-     * The error response obersable for all websocket message errors.
-     */
-    public get errorResponseObservable(): Observable<WebsocketErrorContent> {
-        return this._errorResponseSubject.asObservable();
-    }
-
-    /**
-     * Saves, if the connection is open
-     */
-    private _connectionOpen = false;
-
-    /**
-     * Whether the WebSocket connection is established
-     */
-    public get isConnected(): boolean {
-        return this._connectionOpen;
-    }
-
-    private sendQueueWhileNotConnected: (string | ArrayBuffer)[] = [];
-
-    /**
-     * The websocket.
-     */
-    private websocket: WebSocket | null;
-    private websocketId: string | null;
-
-    /**
-     * Subjects for types of websocket messages. A subscriber can get an Observable by {@function getOberservable}.
-     */
-    private subjects: { [type: string]: Subject<any> } = {};
-
-    /**
-     * Callbacks for a waiting response. If any callback returns true, the message/error will not be propagated with the
-     * responsible subjects for the message type.
-     */
-    private responseCallbacks: {
-        [id: string]: [(val: any) => boolean, (error: WebsocketErrorContent) => boolean];
-    } = {};
-
-    /**
-     * Saves, if the WS Connection should be closed (e.g. after an explicit `close()`). Prohibits
-     * retry connection attempts.
-     */
-    private shouldBeClosed = true;
-
-    /**
-     * Counter for delaying the offline message.
-     */
-    private retryCounter = 0;
-
-    /**
-     * The timeout in the onClose-handler for the next reconnect retry.
-     */
-    private retryTimeout: any = null;
-
-    /**
-     * Constructor that handles the router
-     *
-     * @param zone
-     * @param router
-     * @param openSlidesStatusService
-     * @param offlineService
-     */
-    public constructor(
-        private zone: NgZone,
-        private router: Router,
-        private openSlidesStatusService: OpenSlidesStatusService,
-        private offlineService: OfflineService
-    ) {}
-
-    /**
-     * Creates a new WebSocket connection and handles incomming events.
-     *
-     * Uses NgZone to let all callbacks run in the angular context.
-     */
-    public async connect(changeId: number | null = null, retry: boolean = false): Promise<void> {
-        const websocketId = Math.random().toString(36).substring(7);
-        this.websocketId = websocketId;
-
-        if (this.websocket) {
-            this.websocket.close();
-            this.websocket = null;
-        }
-
-        if (!retry) {
-            this.shouldBeClosed = false;
-        }
-
-        const queryParams: QueryParams = {};
-
-        if (changeId !== null) {
-            queryParams.change_id = changeId;
-        }
-
-        // Create the websocket
-        let socketPath = location.protocol === 'https:' ? 'wss://' : 'ws://';
-        socketPath += window.location.host;
-        if (this.openSlidesStatusService.isPrioritizedClient) {
-            socketPath += '/prioritize';
-        }
-        socketPath += '/ws/';
-        socketPath += formatQueryParams(queryParams);
-
-        this.websocket = new WebSocket(socketPath);
-        this.websocket.binaryType = 'arraybuffer';
-
-        // connection established. If this connect attept was a retry,
-        // The error notice will be removed and the reconnectSubject is published.
-        this.websocket.onopen = (event: Event) => {
-            if (this.websocketId !== websocketId) {
-                return;
-            }
-            this.zone.run(() => {
-                this.retryCounter = 0;
-
-                if (this.shouldBeClosed) {
-                    this.offlineService.goOnline();
-                    return;
-                }
-
-                this._connectionOpen = true;
-                if (retry) {
-                    this.offlineService.goOnline();
-                    this._retryReconnectEvent.emit();
-                } else {
-                    this._noRetryConnectEvent.emit();
-                }
-                this._generalConnectEvent.emit();
-                this.sendQueueWhileNotConnected.forEach(entry => {
-                    this.websocket.send(entry);
-                });
-                this.sendQueueWhileNotConnected = [];
-            });
-        };
-
-        this.websocket.onmessage = (event: MessageEvent) => {
-            if (this.websocketId !== websocketId) {
-                return;
-            }
-            this.zone.run(() => {
-                this.handleMessage(event.data);
-            });
-        };
-
-        this.websocket.onclose = (event: CloseEvent) => {
-            if (this.websocketId !== websocketId) {
-                return;
-            }
-            this.zone.run(() => {
-                this.onclose();
-            });
-        };
-
-        this.websocket.onerror = (event: ErrorEvent) => {
-            if (this.websocketId !== websocketId) {
-                return;
-            }
-            // place for proper error handling and debugging.
-            // Required to get more information about errors
-            this.zone.run(() => {
-                console.warn('WS error event:', event);
-            });
-        };
-    }
-
-    /**
-     * Handles an incomming message.
-     *
-     * @param data The message
-     */
-    private handleMessage(data: string | ArrayBuffer): void {
-        if (data instanceof ArrayBuffer) {
-            const compressedSize = data.byteLength;
-            const decompressedBuffer: Uint8Array = decompress(new Uint8Array(data));
-            console.debug(
-                `Recieved ${compressedSize / 1024} KB (${
-                    decompressedBuffer.byteLength / 1024
-                } KB uncompressed), ratio ${decompressedBuffer.byteLength / compressedSize}`
-            );
-            data = this.arrayBufferToString(decompressedBuffer);
-        }
-
-        const message: IncommingWebsocketMessage = JSON.parse(data);
-        console.debug('Received', message);
-        const type = message.type;
-        const inResponse = message.in_response;
-        const callbacks = this.responseCallbacks[inResponse];
-        if (callbacks) {
-            delete this.responseCallbacks[inResponse];
-        }
-
-        if (type === 'error') {
-            if (!isWebsocketErrorContent(message.content)) {
-                console.error('Websocket error without standard form!', message);
-                return;
-            }
-
-            // Print this to the console.
-            const error = message.content;
-            const errorDescription =
-                Object.keys(WEBSOCKET_ERROR_CODES).find(key => WEBSOCKET_ERROR_CODES[key] === error.code) ||
-                'unknown code';
-            console.error(`Websocket error with code=${error.code} (${errorDescription}):`, error.message);
-
-            // call the error callback, if there is any. If it returns true (means "handled"),
-            // the errorResponseSubject will not be called
-            if (inResponse && callbacks && callbacks[1] && callbacks[1](error)) {
-                return;
-            }
-            this._errorResponseSubject.next(error);
-            return;
-        }
-
-        // Try to fire a response callback directly. If it returnes true, the message is handeled
-        // and not distributed further
-        if (inResponse && callbacks && callbacks[0](message.content)) {
-            return;
-        }
-
-        if (this.subjects[type]) {
-            // Pass the content to the registered subscribers.
-            this.subjects[type].next(message.content);
-        } else {
-            console.warn(
-                `Got unknown websocket message type "${type}" (inResponse: ${inResponse}) with content`,
-                message.content
-            );
-        }
-    }
-
-    /**
-     * Closes the connection error notice
-     */
-    private onclose(): void {
-        if (this.websocket) {
-            this.websocketId = null; // set to null, so now further events will be
-            // registered with the line below.
-            this.websocket.close(); // Cleanup old connection
-            this.websocket = null;
-        }
-        this._connectionOpen = false;
-        // 1000 is a normal close, like the close on logout
-        this._closeEvent.emit();
-        if (!this.shouldBeClosed) {
-            // Do not show the message snackbar on the projector
-            // tests for /projector and /projector/<id>
-            const onProjector = this.router.url.match(/^\/projector(\/[0-9]+\/?)?$/);
-            if (this.retryCounter <= 3) {
-                this.retryCounter++;
-            }
-
-            if (!this.connectionErrorNotice && !onProjector && this.retryCounter > 3) {
-                this.offlineService.goOfflineBecauseConnectionLost();
-            }
-
-            // A random retry timeout between 2000 and 5000 ms.
-            const timeout = Math.floor(Math.random() * 3000 + 2000);
-            this.retryTimeout = setTimeout(() => {
-                this.retryTimeout = null;
-                this.connect(null, true);
-            }, timeout);
-        }
-    }
-
-    public cancelReconnectenRetry(): void {
-        if (this.retryTimeout) {
-            clearTimeout(this.retryTimeout);
-            this.retryTimeout = null;
-        }
-    }
-
-    /**
-     * Closes the websocket connection.
-     */
-    public async close(): Promise<void> {
-        this.shouldBeClosed = true;
-        this.offlineService.goOnline();
-        if (this.websocket) {
-            this.websocket.close();
-            this.websocket = null;
-            await this.closeEvent.pipe(take(1)).toPromise();
-        }
-    }
-
-    /**
-     * Simulates an abnormal close.
-     *
-     * Internally does not set `shouldBeClosed`, so a reconnect is forced.
-     */
-    public simulateAbnormalClose(): void {
-        this.onclose();
-    }
-
-    /**
-     * closes and reopens the connection. If the connection was closed before,
-     * it will be just opened.
-     *
-     * @param options The options for the new connection
-     */
-    public async reconnect(changeId: number | null = null): Promise<void> {
-        await this.close();
-        await this.connect(changeId);
-    }
-
-    /**
-     * Returns an observable for messages of the given type.
-     * @param type the message type
-     */
-    public getOberservable<T>(type: string): Observable<T> {
-        if (!this.subjects[type]) {
-            this.subjects[type] = new Subject<T>();
-        }
-        return this.subjects[type].asObservable();
-    }
-
-    /**
-     * Sends a message to the server with the content and the given type.
-     *
-     * @param type the message type
-     * @param content the actual content
-     * @param success an optional success callback for a response. If it returns true, the message will not be
-     * propagated through the recieve subjects.
-     * @param error an optional error callback for a response. If it returns true, the error will not be propagated
-     * with the error subject.
-     * @param id an optional id for the message. If not given, a random id will be generated and returned.
-     * @returns the message id
-     */
-    public send<T, R>(
-        type: string,
-        content: T,
-        success?: (val: R) => boolean,
-        error?: (error: WebsocketErrorContent) => boolean,
-        id?: string
-    ): string {
-        if (!this.websocket) {
-            return;
-        }
-
-        const message: OutgoingWebsocketMessage = {
-            type: type,
-            content: content,
-            id: id
-        };
-
-        // create message id if not given. Required by the server.
-        if (!message.id) {
-            message.id = '';
-            const possible = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
-            for (let i = 0; i < 8; i++) {
-                message.id += possible.charAt(Math.floor(Math.random() * possible.length));
-            }
-        }
-
-        if (success) {
-            this.responseCallbacks[message.id] = [success, error];
-        }
-
-        const jsonMessage = JSON.stringify(message);
-        const bytesMessage = this.stringToBuffer(jsonMessage);
-
-        const compressedMessage: ArrayBuffer = compress(bytesMessage);
-        const ratio = bytesMessage.byteLength / compressedMessage.byteLength;
-
-        const toSend = ratio > 1 ? compressedMessage : jsonMessage;
-
-        if (this.isConnected) {
-            this.websocket.send(toSend);
-        } else {
-            this.sendQueueWhileNotConnected.push(toSend);
-        }
-
-        return message.id;
-    }
-
-    /**
-     * Sends a message and waits for the response
-     *
-     * @param type the message type
-     * @param content the actual content
-     * @param id an optional id for the message. If not given, a random id will be generated and returned.
-     */
-    public sendAndGetResponse<T, R>(type: string, content: T, id?: string): Promise<R> {
-        return new Promise<R>((resolve, reject) => {
-            this.send<T, R>(
-                type,
-                content,
-                val => {
-                    resolve(val);
-                    return true;
-                },
-                val => {
-                    reject(val);
-                    return true;
-                },
-                id
-            );
-        });
-    }
-
-    /**
-     * Converts an ArrayBuffer to a String.
-     *
-     * @param buffer - Buffer to convert
-     * @returns String
-     */
-    private arrayBufferToString(buffer: Uint8Array): string {
-        return Array.from(buffer)
-            .map(code => String.fromCharCode(code))
-            .join('');
-    }
-
-    /**
-     * Converts a String to an ArrayBuffer.
-     *
-     * @param str - String to convert.
-     * @returns bufferView.
-     */
-    private stringToBuffer(str: string): Uint8Array {
-        const bufferView = new Uint8Array();
-        for (let i = 0; i < str.length; i++) {
-            bufferView[i] = str.charCodeAt(i);
-        }
-        return bufferView;
-    }
-}

client/src/app/core/definitions/autoupdate-format.ts

@@ -0,0 +1,43 @@
+export interface AutoupdateFormat {
+    /**
+     * All changed (and created) items as their full/restricted data grouped by their collection.
+     */
+    changed: {
+        [collectionString: string]: object[];
+    };
+
+    /**
+     * All deleted items (by id) grouped by their collection.
+     */
+    deleted: {
+        [collectionString: string]: number[];
+    };
+
+    /**
+     * The lower change id bond for this autoupdate
+     */
+    from_change_id: number;
+
+    /**
+     * The upper change id bound for this autoupdate
+     */
+    to_change_id: number;
+
+    /**
+     * Flag, if this autoupdate contains all data. If so, the DS needs to be resetted.
+     */
+    all_data: boolean;
+}
+
+export function isAutoupdateFormat(obj: any): obj is AutoupdateFormat {
+    const format = obj as AutoupdateFormat;
+    return (
+        obj &&
+        typeof obj === 'object' &&
+        format.changed !== undefined &&
+        format.deleted !== undefined &&
+        format.from_change_id !== undefined &&
+        format.to_change_id !== undefined &&
+        format.all_data !== undefined
+    );
+}

client/src/app/core/definitions/http-methods.ts

@@ -0,0 +1,10 @@
+/**
+ * Enum for different HTTPMethods
+ */
+export enum HTTPMethod {
+    GET = 'get',
+    POST = 'post',
+    PUT = 'put',
+    PATCH = 'patch',
+    DELETE = 'delete'
+}

client/src/app/core/promises/deferred.ts

@@ -14,16 +14,16 @@
  * ```
  */
 export class Deferred<T = void> extends Promise<T> {
-    /**
-     * The promise to wait for
-     */
-    public readonly promise: Promise<T>;
-
     /**
      * custom resolve function
      */
     private _resolve: (val?: T) => void;
 
+    private _wasResolved;
+    public get wasResolved(): boolean {
+        return this._wasResolved;
+    }
+
     /**
      * Creates the promise and overloads the resolve function
      */
@@ -32,6 +32,7 @@ export class Deferred<T = void> extends Promise<T> {
         super(resolve => {
             preResolve = resolve;
         });
+        this._wasResolved = false;
         this._resolve = preResolve;
     }
 
@@ -40,5 +41,6 @@ export class Deferred<T = void> extends Promise<T> {
      */
     public resolve(val?: T): void {
         this._resolve(val);
+        this._wasResolved = true;
     }
 }

client/src/app/core/promises/sleep.ts

@@ -0,0 +1,10 @@
+/**
+ * Wraps a promise and let it reject after the given timeout (in ms), if it was
+ * not resolved before this timeout.
+ *
+ * @param delay The time to sleep in miliseconds
+ * @returns a new Promise
+ */
+export function SleepPromise(delay: number): Promise<void> {
+    return new Promise((resolve, _) => setTimeout(resolve, delay));
+}

client/src/app/core/rxjs/trailing-throttle-time.ts

@@ -0,0 +1,6 @@
+import { MonoTypeOperatorFunction } from 'rxjs';
+import { throttleTime } from 'rxjs/operators';
+
+export function trailingThrottleTime<T = unknown>(time: number): MonoTypeOperatorFunction<T> {
+    return throttleTime<T>(time, undefined, { leading: false, trailing: true });
+}

client/src/app/core/ui-services/banner.service.ts

@@ -1,7 +1,10 @@
 import { Injectable } from '@angular/core';
 
+import { marker as _ } from '@biesbjerg/ngx-translate-extract-marker';
 import { BehaviorSubject } from 'rxjs';
 
+import { OfflineBroadcastService } from '../core-services/offline-broadcast.service';
+
 export interface BannerDefinition {
     type?: string;
     class?: string;
@@ -20,8 +23,26 @@ export interface BannerDefinition {
     providedIn: 'root'
 })
 export class BannerService {
+    private offlineBannerDefinition: BannerDefinition = {
+        text: _('Offline mode'),
+        icon: 'cloud_off'
+    };
+
     public activeBanners: BehaviorSubject<BannerDefinition[]> = new BehaviorSubject<BannerDefinition[]>([]);
 
+    public constructor(/*translate: TranslateService, */ offlineBroadcastService: OfflineBroadcastService) {
+        /*translate.onLangChange.subscribe(() => {
+            this.offlineBannerDefinition.text = translate.instant(this.offlineBannerDefinition.text);
+        });*/
+        offlineBroadcastService.isOfflineObservable.subscribe(offline => {
+            if (offline) {
+                this.addBanner(this.offlineBannerDefinition);
+            } else {
+                this.removeBanner(this.offlineBannerDefinition);
+            }
+        });
+    }
+
     /**
      * Adds a banner to the list of active banners. Skip the banner if it's already in the list
      * @param toAdd the banner to add

client/src/app/core/ui-services/count-users.service.ts

@@ -42,24 +42,24 @@ export class CountUsersService {
     public constructor(private notifyService: NotifyService, operator: OperatorService) {
         // Listen for requests to send an answer.
         this.notifyService.getMessageObservable<CountUserRequest>(REQUEST_NAME).subscribe(request => {
-            if (request.content.token) {
+            if (request.message.token) {
                 this.notifyService.sendToChannels<CountUserResponse>(
                     RESPONSE_NAME,
                     {
-                        token: request.content.token,
+                        token: request.message.token,
                         data: {
                             userId: this.currentUserId
                         }
                     },
-                    request.senderChannelName
+                    request.sender_channel_id
                 );
             }
         });
 
         // Listen for responses and distribute them through `activeCounts`
         this.notifyService.getMessageObservable<CountUserResponse>(RESPONSE_NAME).subscribe(response => {
-            if (response.content.data && response.content.token && this.activeCounts[response.content.token]) {
+            if (response.message.data && response.message.token && this.activeCounts[response.message.token]) {
-                this.activeCounts[response.content.token].next(response.content.data);
+                this.activeCounts[response.message.token].next(response.message.data);
             }
         });
 

client/src/app/core/ui-services/overlay.service.ts

@@ -7,7 +7,7 @@ import { distinctUntilChanged } from 'rxjs/operators';
 import { largeDialogSettings } from 'app/shared/utils/dialog-settings';
 import { SuperSearchComponent } from 'app/site/common/components/super-search/super-search.component';
 import { DataStoreUpgradeService } from '../core-services/data-store-upgrade.service';
-import { OfflineService } from '../core-services/offline.service';
+import { OfflineBroadcastService } from '../core-services/offline-broadcast.service';
 import { OpenSlidesService } from '../core-services/openslides.service';
 import { OperatorService } from '../core-services/operator.service';
 
@@ -59,7 +59,7 @@ export class OverlayService {
         private operator: OperatorService,
         OpenSlides: OpenSlidesService,
         upgradeService: DataStoreUpgradeService,
-        offlineService: OfflineService
+        offlineBroadcastService: OfflineBroadcastService
     ) {
         // Subscribe to the current user.
         operator.getViewUserObservable().subscribe(user => {
@@ -79,13 +79,10 @@ export class OverlayService {
             this.checkConnection();
         });
         // Subscribe to check if we are offline
-        offlineService
+        offlineBroadcastService.isOfflineObservable.pipe(distinctUntilChanged()).subscribe(offline => {
-            .isOffline()
+            this.isOffline = offline;
-            .pipe(distinctUntilChanged())
+            this.checkConnection();
-            .subscribe(offline => {
+        });
-                this.isOffline = offline;
-                this.checkConnection();
-            });
     }
 
     /**

client/src/app/shared/components/copyright-sign/copyright-sign.component.ts

@@ -211,15 +211,15 @@ export class C4DialogComponent implements OnInit, OnDestroy {
         search: {
             recievedSearchRequest: {
                 handle: (notify: NotifyResponse<{ name: string }>) => {
-                    this.replyChannel = notify.senderChannelName;
+                    this.replyChannel = notify.sender_channel_id;
-                    this.partnerName = notify.content.name;
+                    this.partnerName = notify.message.name;
                     return 'waitForResponse';
                 }
             },
             recievedSearchResponse: {
                 handle: (notify: NotifyResponse<{ name: string }>) => {
-                    this.replyChannel = notify.senderChannelName;
+                    this.replyChannel = notify.sender_channel_id;
-                    this.partnerName = notify.content.name;
+                    this.partnerName = notify.message.name;
                     // who starts?
                     const startPlayer = Math.random() < 0.5 ? Player.thisPlayer : Player.partner;
                     const startPartner: boolean = startPlayer === Player.partner;
@@ -232,10 +232,10 @@ export class C4DialogComponent implements OnInit, OnDestroy {
         waitForResponse: {
             recievedACK: {
                 handle: (notify: NotifyResponse<{}>) => {
-                    if (notify.senderChannelName !== this.replyChannel) {
+                    if (notify.sender_channel_id !== this.replyChannel) {
                         return null;
                     }
-                    return notify.content ? 'myTurn' : 'foreignTurn';
+                    return notify.message ? 'myTurn' : 'foreignTurn';
                 }
             },
             waitTimeout: {
@@ -243,7 +243,7 @@ export class C4DialogComponent implements OnInit, OnDestroy {
             },
             recievedRagequit: {
                 handle: (notify: NotifyResponse<{}>) => {
-                    return notify.senderChannelName === this.replyChannel ? 'search' : null;
+                    return notify.sender_channel_id === this.replyChannel ? 'search' : null;
                 }
             }
         },
@@ -270,10 +270,10 @@ export class C4DialogComponent implements OnInit, OnDestroy {
         foreignTurn: {
             recievedTurn: {
                 handle: (notify: NotifyResponse<{ col: number }>) => {
-                    if (notify.senderChannelName !== this.replyChannel) {
+                    if (notify.sender_channel_id !== this.replyChannel) {
                         return null;
                     }
-                    const col: number = notify.content.col;
+                    const col: number = notify.message.col;
                     if (!this.colFree(col)) {
                         return null;
                     }
@@ -455,7 +455,7 @@ export class C4DialogComponent implements OnInit, OnDestroy {
      */
     public enter_waitForResponse(): void {
         this.caption = 'Wait for response...';
-        this.notifyService.send('c4_search_response', { name: this.getPlayerName() });
+        this.notifyService.sendToChannels('c4_search_response', { name: this.getPlayerName() }, this.replyChannel);
         if (this.waitTimout) {
             clearTimeout(<any>this.waitTimout);
         }

client/src/app/shared/components/list-view-table/list-view-table.component.ts

@@ -28,6 +28,7 @@ import { ViewportService } from 'app/core/ui-services/viewport.service';
 import { BaseProjectableViewModel } from 'app/site/base/base-projectable-view-model';
 import { BaseViewModel } from 'app/site/base/base-view-model';
 import { BaseViewModelWithContentObject } from 'app/site/base/base-view-model-with-content-object';
+import { isProjectable } from 'app/site/base/projectable';
 
 export interface CssClassDefinition {
     [key: string]: boolean;
@@ -458,8 +459,8 @@ export class ListViewTableComponent<V extends BaseViewModel | BaseViewModelWithC
     }
 
     public isElementProjected = (context: PblNgridRowContext<V>) => {
-        const model = context.$implicit as V;
+        const projectableViewModel = this.getProjectable(context.$implicit as V);
-        if (this.allowProjector && this.projectorService.isProjected(this.getProjectable(model))) {
+        if (projectableViewModel && this.allowProjector && this.projectorService.isProjected(projectableViewModel)) {
             return 'projected';
         }
     };
@@ -578,8 +579,10 @@ export class ListViewTableComponent<V extends BaseViewModel | BaseViewModelWithC
      * @param viewModel The model of the table
      * @returns a view model that can be projected
      */
-    public getProjectable(viewModel: V): BaseProjectableViewModel {
+    public getProjectable(viewModel: V): BaseProjectableViewModel | null {
-        return (viewModel as BaseViewModelWithContentObject)?.contentObject ?? viewModel;
+        const actualViewModel: BaseProjectableViewModel =
+            (viewModel as BaseViewModelWithContentObject)?.contentObject ?? viewModel;
+        return isProjectable(actualViewModel) ? actualViewModel : null;
     }
 
     /**

client/src/app/shared/components/projector/projector.component.ts

@@ -5,7 +5,7 @@ import { TranslateService } from '@ngx-translate/core';
 import { Subject, Subscription } from 'rxjs';
 
 import { BaseComponent } from 'app/base.component';
-import { OfflineService } from 'app/core/core-services/offline.service';
+import { OfflineBroadcastService } from 'app/core/core-services/offline-broadcast.service';
 import { ProjectorDataService, SlideData } from 'app/core/core-services/projector-data.service';
 import { ProjectorRepositoryService } from 'app/core/repositories/projector/projector-repository.service';
 import { ConfigService } from 'app/core/ui-services/config.service';
@@ -166,7 +166,7 @@ export class ProjectorComponent extends BaseComponent implements OnDestroy {
         private projectorDataService: ProjectorDataService,
         private projectorRepository: ProjectorRepositoryService,
         private configService: ConfigService,
-        private offlineService: OfflineService,
+        private offlineBroadcastService: OfflineBroadcastService,
         private elementRef: ElementRef
     ) {
         super(titleService, translate);
@@ -207,7 +207,9 @@ export class ProjectorComponent extends BaseComponent implements OnDestroy {
             }
         });
 
-        this.offlineSubscription = this.offlineService.isOffline().subscribe(isOffline => (this.isOffline = isOffline));
+        this.offlineSubscription = this.offlineBroadcastService.isOfflineObservable.subscribe(
+            isOffline => (this.isOffline = isOffline)
+        );
     }
 
     /**

client/src/app/site/common/components/start/start.component.html

@@ -13,6 +13,7 @@
 </os-head-bar>
 
 <mat-card class="spacer-bottom-60" [ngClass]="isEditing ? 'os-form-card' : 'os-card'">
+<mat-card [ngClass]="isEditing ? 'os-form-card' : 'os-card'">
     <ng-container *ngIf="!isEditing">
         <div class="app-content">
             <h1>{{ startContent.general_event_welcome_title | translate }}</h1>

client/src/app/site/motions/modules/motion-detail/components/motion-detail/motion-detail.component.ts

@@ -1639,7 +1639,7 @@ export class MotionDetailComponent extends BaseViewComponentDirective implements
      */
     private listenToEditNotification(): Subscription {
         return this.notifyService.getMessageObservable(this.NOTIFICATION_EDIT_MOTION).subscribe(message => {
-            const content = <MotionEditNotification>message.content;
+            const content = <MotionEditNotification>message.message;
             if (this.operator.viewUser.id !== content.senderId && content.motionId === this.motion.id) {
                 let warning = '';
 
@@ -1656,7 +1656,7 @@ export class MotionDetailComponent extends BaseViewComponentDirective implements
                         if (content.type === MotionEditNotificationType.TYPE_BEGIN_EDITING_MOTION) {
                             this.sendEditNotification(
                                 MotionEditNotificationType.TYPE_ALSO_EDITING_MOTION,
-                                message.senderUserId
+                                message.sender_user_id
                             );
                         }
                         break;

client/src/app/site/polls/components/base-poll-dialog.component.ts

@@ -58,8 +58,6 @@ export abstract class BasePollDialogComponent<
             votes: this.getVoteData(),
             publish_immediately: this.publishImmediately
         };
-        console.log('answer: ', answer);
-
         this.dialogRef.close(answer);
     }
 

client/src/app/site/site.component.ts

@@ -10,7 +10,7 @@ import { TranslateService } from '@ngx-translate/core';
 import { filter } from 'rxjs/operators';
 
 import { navItemAnim } from '../shared/animations';
-import { OfflineService } from 'app/core/core-services/offline.service';
+import { OfflineBroadcastService } from 'app/core/core-services/offline-broadcast.service';
 import { OverlayService } from 'app/core/ui-services/overlay.service';
 import { UpdateService } from 'app/core/ui-services/update.service';
 import { BaseComponent } from '../base.component';
@@ -84,7 +84,7 @@ export class SiteComponent extends BaseComponent implements OnInit {
     public constructor(
         title: Title,
         protected translate: TranslateService,
-        offlineService: OfflineService,
+        offlineBroadcastService: OfflineBroadcastService,
         private updateService: UpdateService,
         private router: Router,
         public operator: OperatorService,
@@ -99,7 +99,7 @@ export class SiteComponent extends BaseComponent implements OnInit {
         super(title, translate);
         overlayService.showSpinner(translate.instant('Loading data. Please wait ...'));
 
-        offlineService.isOffline().subscribe(offline => {
+        offlineBroadcastService.isOfflineObservable.subscribe(offline => {
             this.isOffline = offline;
         });
 

dc-dev.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+cd "$(dirname $0)"
+docker-compose -f docker/docker-compose.dev.yml $@

docker/.env

@@ -17,10 +17,14 @@ DEFAULT_DOCKER_REGISTRY=
 
 # Docker Images
 # -------------
+DOCKER_OPENSLIDES_HAPROXY_NAME=
+DOCKER_OPENSLIDES_HAPROXY_TAG=
 DOCKER_OPENSLIDES_BACKEND_NAME=
 DOCKER_OPENSLIDES_BACKEND_TAG=
 DOCKER_OPENSLIDES_FRONTEND_NAME=
 DOCKER_OPENSLIDES_FRONTEND_TAG=
+DOCKER_OPENSLIDES_AUTOUPDATE_NAME=
+DOCKER_OPENSLIDES_AUTOUPDATE_TAG=
 
 # Database
 # --------
@@ -37,6 +41,7 @@ PGBOUNCER_PLACEMENT_CONSTR=
 # -------------------
 OPENSLIDES_BACKEND_SERVICE_REPLICAS=
 OPENSLIDES_FRONTEND_SERVICE_REPLICAS=
+OPENSLIDES_AUTOUPDATE_SERVICE_REPLICAS=
 REDIS_RO_SERVICE_REPLICAS=
 MEDIA_SERVICE_REPLICAS=
 

docker/build.sh

@@ -6,7 +6,9 @@ declare -A TARGETS
 TARGETS=(
   [client]="$(dirname "${BASH_SOURCE[0]}")/../client/docker/"
   [server]="$(dirname "${BASH_SOURCE[0]}")/../server/docker/"
-  [media-service]="https://github.com/OpenSlides/openslides-media-service.git"
+  [haproxy]="$(dirname "${BASH_SOURCE[0]}")/../haproxy/"
+  [autoupdate]="$(dirname "${BASH_SOURCE[0]}")/../autoupdate/"
+  [media]="https://github.com/OpenSlides/openslides-media-service.git"
   [pgbouncer]="https://github.com/OpenSlides/openslides-docker-compose.git#:pgbouncer"
   [postfix]="https://github.com/OpenSlides/openslides-docker-compose.git#:postfix"
   [repmgr]="https://github.com/OpenSlides/openslides-docker-compose.git#:repmgr"
@@ -17,7 +19,7 @@ DOCKER_TAG="latest"
 CONFIG="/etc/osinstancectl"
 OPTIONS=()
 BUILT_IMAGES=()
-DEFAULT_TARGETS=(server client)
+DEFAULT_TARGETS=(server client haproxy autoupdate)
 
 usage() {
   cat << EOF

docker/docker-compose.dev.yml

@@ -0,0 +1,47 @@
+version: "3"
+services:
+  client:
+    image: os3-client-dev
+    build:
+        context: ../client
+        dockerfile: docker/Dockerfile.dev
+    volumes:
+        - ../client:/app
+
+  server:
+    image: os3-server-dev
+    user: $UID:$GID
+    build:
+        context: ../server
+        dockerfile: docker/Dockerfile.dev
+    volumes:
+        - ../server:/app
+    depends_on:
+      - redis
+
+  autoupdate:
+    image: os3-autoupdate-dev
+    environment:
+        - MESSAGE_BUS_HOST=redis
+        - MESSAGE_BUS_PORT=6379
+        - WORKER_HOST=server
+        - WORKER_PORT=8000
+    depends_on:
+        - server
+        - redis
+    volumes:
+      - ../autoupdate/cmd:/root/cmd
+      - ../autoupdate/internal:/root/internal
+  redis:
+    image: redis:latest
+
+  haproxy:
+    image: os3-haproxy-dev
+    volumes:
+      - ../haproxy/src:/usr/local/etc/haproxy
+    depends_on:
+      - client
+      - server
+      - autoupdate
+    ports:
+      - "8000:8000"

docker/docker-compose.yml.m4

@@ -13,12 +13,22 @@ define(`read_env', `esyscmd(`printf "\`%s'" "$$1"')')
 dnl return env variable if set; otherwise, return given alternative value
 define(`ifenvelse', `ifelse(read_env(`$1'),, `$2', read_env(`$1'))')
 
+define(`HAPROXY_IMAGE',
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_NAME', openslides-haproxy):dnl
+ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_TAG', latest))
 define(`BACKEND_IMAGE',
-ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides/openslides-server):dnl
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides-server):dnl
 ifenvelse(`DOCKER_OPENSLIDES_BACKEND_TAG', latest))
 define(`FRONTEND_IMAGE',
-ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides/openslides-client):dnl
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides-client):dnl
 ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_TAG', latest))
+define(`AUTOUPDATE_IMAGE',
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_NAME', openslides-autoupdate):dnl
+ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_TAG', latest))
 
 define(`PRIMARY_DB', `ifenvelse(`PGNODE_REPMGR_PRIMARY', pgnode1)')
 
@@ -41,11 +51,9 @@ x-osserver:
   &default-osserver
   image: BACKEND_IMAGE
   networks:
-    - front
     - back
   restart: always
 x-osserver-env: &default-osserver-env
-    AMOUNT_REPLICAS: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1)
     AUTOUPDATE_DELAY: ifenvelse(`AUTOUPDATE_DELAY', 1)
     DEMO_USERS: "ifenvelse(`DEMO_USERS',)"
     CONNECTION_POOL_LIMIT: ifenvelse(`CONNECTION_POOL_LIMIT', 100)
@@ -54,7 +62,6 @@ x-osserver-env: &default-osserver-env
     DATABASE_PORT: ifenvelse(`DATABASE_PORT', 5432)
     DATABASE_USER: "ifenvelse(`DATABASE_USER', openslides)"
     DEFAULT_FROM_EMAIL: "ifenvelse(`DEFAULT_FROM_EMAIL', noreply@example.com)"
-    DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
     EMAIL_HOST: "ifenvelse(`EMAIL_HOST', postfix)"
     EMAIL_HOST_PASSWORD: "ifenvelse(`EMAIL_HOST_PASSWORD',)"
     EMAIL_HOST_USER: "ifenvelse(`EMAIL_HOST_USER',)"
@@ -69,13 +76,11 @@ x-osserver-env: &default-osserver-env
     JITSI_ROOM_PASSWORD: "ifenvelse(`JITSI_ROOM_PASSWORD',)"
     JITSI_ROOM_NAME: "ifenvelse(`JITSI_ROOM_NAME',)"
     OPENSLIDES_LOG_LEVEL: "ifenvelse(`OPENSLIDES_LOG_LEVEL', INFO)"
-    REDIS_CHANNLES_HOST: "ifenvelse(`REDIS_CHANNLES_HOST', redis-channels)"
+    DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
-    REDIS_CHANNLES_PORT: ifenvelse(`REDIS_CHANNLES_PORT', 6379)
     REDIS_HOST: "ifenvelse(`REDIS_HOST', redis)"
     REDIS_PORT: ifenvelse(`REDIS_PORT', 6379)
     REDIS_SLAVE_HOST: "ifenvelse(`REDIS_SLAVE_HOST', redis-slave)"
     REDIS_SLAVE_PORT: ifenvelse(`REDIS_SLAVE_PORT', 6379)
-    REDIS_SLAVE_WAIT_TIMEOUT: ifenvelse(`REDIS_SLAVE_WAIT_TIMEOUT', 10000)
     RESET_PASSWORD_VERBOSE_ERRORS: "ifenvelse(`RESET_PASSWORD_VERBOSE_ERRORS', False)"
 x-pgnode: &default-pgnode
   image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-repmgr:latest
@@ -90,15 +95,27 @@ x-pgnode-env: &default-pgnode-env
   REPMGR_WAL_ARCHIVE: "ifenvelse(`PGNODE_WAL_ARCHIVING', on)"
 
 services:
+  haproxy:
+    image: HAPROXY_IMAGE
+    depends_on:
+      - server
+      - client
+      - autoupdate
+      - media
+    networks:
+      - front
+      - back
+    ports:
+      - "127.0.0.1:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):8000"
+
   server:
     << : *default-osserver
     # Below is the default command.  You can uncomment it to override the
     # number of workers, for example:
-    # command: "gunicorn -w 8 --preload -b 0.0.0.0:8000
+    # command: "gunicorn -w 8 --preload -b 0.0.0.0:8000 openslides.wsgi"
-    #   -k uvicorn.workers.UvicornWorker openslides.asgi:application"
     #
     # Uncomment the following line to use daphne instead of gunicorn:
-    # command: "daphne -b 0.0.0.0 -p 8000 openslides.asgi:application"
+    # command: "daphne -b 0.0.0.0 -p 8000 openslides.wsgi"
     depends_on:
       - server-setup
     environment:
@@ -127,17 +144,24 @@ services:
       - pgbouncer
       - redis
       - redis-slave
-      - redis-channels
 
   client:
     image: FRONTEND_IMAGE
     restart: always
-    depends_on:
-      - server
     networks:
-      - front
+      - back
-    ports:
+
-      - "127.0.0.1:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):80"
+  autoupdate:
+    image: AUTOUPDATE_IMAGE
+    restart: always
+    depends_on:
+      - redis
+      - server
+    environment:
+      MESSAGE_BUS_HOST: redis
+      WORKER_HOST: server
+    networks:
+      - back
 
   pgnode1:
     << : *default-pgnode
@@ -189,9 +213,7 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
     image: redis:alpine
     restart: always
     networks:
-      back:
+      - back
-        aliases:
-          - rediscache
   redis-slave:
     image: redis:alpine
     restart: always
@@ -199,18 +221,11 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
     depends_on:
       - redis
     networks:
-      back:
+      - back
-        aliases:
-          - rediscache-slave
     ifelse(read_env(`REDIS_RO_SERVICE_REPLICAS'),,,deploy:
       replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1))
-  redis-channels:
-    image: redis:alpine
-    restart: always
-    networks:
-      back:
   media:
-    image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media-service:latest
+    image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media:latest
     environment:
       - CHECK_REQUEST_URL=server:8000/check-media/
       - CACHE_SIZE=ifenvelse(`CACHE_SIZE', 10)
@@ -218,8 +233,7 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
       - CACHE_DATA_MAX_SIZE_KB=ifenvelse(`CACHE_DATA_MAX_SIZE_KB', 10240)
     restart: always
     networks:
-      front:
+      - back
-      back:
     # Override command to run more workers per task
     # command: ["gunicorn", "-w", "4", "--preload", "-b",
     #   "0.0.0.0:8000", "src.mediaserver:app"]

docker/docker-stack.yml.m4

@@ -13,12 +13,22 @@ define(`read_env', `esyscmd(`printf "\`%s'" "$$1"')')
 dnl return env variable if set; otherwise, return given alternative value
 define(`ifenvelse', `ifelse(read_env(`$1'),, `$2', read_env(`$1'))')
 
+define(`HAPROXY_IMAGE',
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_NAME', openslides-haproxy):dnl
+ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_TAG', latest))
 define(`BACKEND_IMAGE',
-ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides/openslides-server):dnl
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides-server):dnl
 ifenvelse(`DOCKER_OPENSLIDES_BACKEND_TAG', latest))
 define(`FRONTEND_IMAGE',
-ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides/openslides-client):dnl
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides-client):dnl
 ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_TAG', latest))
+define(`AUTOUPDATE_IMAGE',
+ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
+ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_NAME', openslides-autoupdate):dnl
+ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_TAG', latest))
 
 define(`PRIMARY_DB', `ifenvelse(`PGNODE_REPMGR_PRIMARY', pgnode1)')
 
@@ -41,10 +51,8 @@ x-osserver:
   &default-osserver
   image: BACKEND_IMAGE
   networks:
-    - front
     - back
 x-osserver-env: &default-osserver-env
-    AMOUNT_REPLICAS: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 3)
     AUTOUPDATE_DELAY: ifenvelse(`AUTOUPDATE_DELAY', 1)
     DEMO_USERS: "ifenvelse(`DEMO_USERS',)"
     CONNECTION_POOL_LIMIT: ifenvelse(`CONNECTION_POOL_LIMIT', 100)
@@ -53,7 +61,6 @@ x-osserver-env: &default-osserver-env
     DATABASE_PORT: ifenvelse(`DATABASE_PORT', 5432)
     DATABASE_USER: "ifenvelse(`DATABASE_USER', openslides)"
     DEFAULT_FROM_EMAIL: "ifenvelse(`DEFAULT_FROM_EMAIL', noreply@example.com)"
-    DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
     EMAIL_HOST: "ifenvelse(`EMAIL_HOST', postfix)"
     EMAIL_HOST_PASSWORD: "ifenvelse(`EMAIL_HOST_PASSWORD',)"
     EMAIL_HOST_USER: "ifenvelse(`EMAIL_HOST_USER',)"
@@ -68,13 +75,11 @@ x-osserver-env: &default-osserver-env
     JITSI_ROOM_PASSWORD: "ifenvelse(`JITSI_ROOM_PASSWORD',)"
     JITSI_ROOM_NAME: "ifenvelse(`JITSI_ROOM_NAME',)"
     OPENSLIDES_LOG_LEVEL: "ifenvelse(`OPENSLIDES_LOG_LEVEL', INFO)"
-    REDIS_CHANNLES_HOST: "ifenvelse(`REDIS_CHANNLES_HOST', redis-channels)"
+    DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
-    REDIS_CHANNLES_PORT: ifenvelse(`REDIS_CHANNLES_PORT', 6379)
     REDIS_HOST: "ifenvelse(`REDIS_HOST', redis)"
     REDIS_PORT: ifenvelse(`REDIS_PORT', 6379)
     REDIS_SLAVE_HOST: "ifenvelse(`REDIS_SLAVE_HOST', redis-slave)"
     REDIS_SLAVE_PORT: ifenvelse(`REDIS_SLAVE_PORT', 6379)
-    REDIS_SLAVE_WAIT_TIMEOUT: ifenvelse(`REDIS_SLAVE_WAIT_TIMEOUT', 10000)
     RESET_PASSWORD_VERBOSE_ERRORS: "ifenvelse(`RESET_PASSWORD_VERBOSE_ERRORS', False)"
 x-pgnode: &default-pgnode
   image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-repmgr:latest
@@ -90,6 +95,18 @@ x-pgnode-env: &default-pgnode-env
   REPMGR_WAL_ARCHIVE: "ifenvelse(`PGNODE_WAL_ARCHIVING', on)"
 
 services:
+  haproxy:
+    image: HAPROXY_IMAGE
+    networks:
+      - front
+      - back
+    ports:
+      - "0.0.0.0:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):8000"
+    deploy:
+      restart_policy:
+        condition: on-failure
+        delay: 5s
+
   server:
     << : *default-osserver
     # Below is the default command.  You can uncomment it to override the
@@ -128,15 +145,26 @@ services:
   client:
     image: FRONTEND_IMAGE
     networks:
-      - front
+      - back
-    ports:
-      - "0.0.0.0:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):80"
     deploy:
       replicas: ifenvelse(`OPENSLIDES_FRONTEND_SERVICE_REPLICAS', 1)
       restart_policy:
         condition: on-failure
         delay: 5s
 
+  autoupdate:
+    image: AUTOUPDATE_IMAGE
+    environment:
+      MESSAGE_BUS_HOST: redis
+      WORKER_HOST: server
+    networks:
+      - back
+    deploy:
+      replicas: ifenvelse(`OPENSLIDES_AUTOUPDATE_SERVICE_REPLICAS', 1)
+      restart_policy:
+        condition: on-failure
+        delay: 5s
+
   pgnode1:
     << : *default-pgnode
     environment:
@@ -206,9 +234,7 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
   redis:
     image: redis:alpine
     networks:
-      back:
+      - back
-        aliases:
-          - rediscache
     deploy:
       replicas: 1
       restart_policy:
@@ -218,38 +244,26 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
     image: redis:alpine
     command: ["redis-server", "--save", "", "--slaveof", "redis", "6379"]
     networks:
-      back:
+      - back
-        aliases:
-          - rediscache-slave
     deploy:
-      replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 3)
+      replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1)
-      restart_policy:
-        condition: on-failure
-        delay: 5s
-  redis-channels:
-    image: redis:alpine
-    networks:
-      back:
-    deploy:
-      replicas: 1
       restart_policy:
         condition: on-failure
         delay: 5s
   media:
-    image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media-service:latest
+    image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media:latest
     environment:
       - CHECK_REQUEST_URL=server:8000/check-media/
       - CACHE_SIZE=ifenvelse(`CACHE_SIZE', 10)
       - CACHE_DATA_MIN_SIZE_KB=ifenvelse(`CACHE_DATA_MIN_SIZE_KB', 0)
       - CACHE_DATA_MAX_SIZE_KB=ifenvelse(`CACHE_DATA_MAX_SIZE_KB', 10240)
     deploy:
-      replicas: ifenvelse(`MEDIA_SERVICE_REPLICAS', 8)
+      replicas: ifenvelse(`MEDIA_SERVICE_REPLICAS', 2)
       restart_policy:
         condition: on-failure
         delay: 10s
     networks:
-      front:
+      - back
-      back:
     # Override command to run more workers per task
     # command: ["gunicorn", "-w", "4", "--preload", "-b",
     #   "0.0.0.0:8000", "src.mediaserver:app"]

haproxy/Dockerfile

@@ -0,0 +1,5 @@
+FROM haproxy:2.0-alpine
+COPY src/haproxy.common.cfg /usr/local/etc/haproxy/haproxy.common.cfg
+COPY src/haproxy.prod.cfg /usr/local/etc/haproxy/haproxy.prod.cfg
+COPY src/combined.pem /usr/local/etc/haproxy/combined.pem
+CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.common.cfg", "-f", "/usr/local/etc/haproxy/haproxy.prod.cfg"]

haproxy/Dockerfile.dev

@@ -0,0 +1,5 @@
+FROM haproxy:2.0-alpine
+COPY src/haproxy.common.cfg /usr/local/etc/haproxy/haproxy.common.cfg
+COPY src/haproxy.dev.cfg /usr/local/etc/haproxy/haproxy.dev.cfg
+COPY src/combined.pem /usr/local/etc/haproxy/combined.pem
+CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.common.cfg", "-f", "/usr/local/etc/haproxy/haproxy.dev.cfg"]

haproxy/Makefile

@@ -0,0 +1,3 @@
+build-dev:
+	./prepare-cert.sh
+	docker build -t os3-haproxy-dev -f Dockerfile.dev .

haproxy/build.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+cd "$(dirname "${BASH_SOURCE[0]}")"
+./prepare-cert.sh
+docker build --tag "${img:-openslides/openslides-${service_name}:latest}" \
+  --pull "${OPTIONS[@]}" .

haproxy/prepare-cert.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+set -e
+cd "$(dirname "$0")"
+
+# check, if we already generated a cert
+combined="src/combined.pem"
+
+if [[ ! -f $combined ]]; then
+    echo "Creating certificates..."
+    cd src
+    if type 2>&1 >/dev/null openssl ; then
+      echo "Using openssl to generate a certificate."
+      echo "You will need to accept an security exception for the"
+      echo "generated certificate in your browser manually."
+      openssl req -x509 -newkey rsa:4096 -nodes -days 3650 \
+              -subj "/C=DE/O=Selfsigned Test/CN=localhost" \
+              -keyout localhost-key.pem -out localhost.pem
+    else
+      echo >&2 "FATAL: No valid certificate generation tool found!"
+      exit -1
+    fi
+    cat localhost.pem localhost-key.pem > combined.pem
+    echo "done"
+else
+    echo "Certificate exists."
+fi

haproxy/src/haproxy.common.cfg

@@ -0,0 +1,31 @@
+global
+    log stdout  format raw  local0  debug
+
+defaults
+    option http-use-htx
+    option dontlognull
+    timeout connect 3s
+    timeout client 10s
+    timeout client-fin 10s
+    timeout server 10s
+    timeout server-fin 10s
+    timeout check 2s
+    timeout tunnel 10s
+    timeout queue 2s
+    log global
+    option httplog
+
+resolvers docker_resolver
+    nameserver dns 127.0.0.11:53
+
+backend backend_server
+    mode http
+    # Do not pass the auth-header from /stats to OS. It confuses the server...
+    http-request del-header authorization
+    timeout server 4m
+    server server server:8000 resolvers docker_resolver check alpn http/1.1
+
+backend backend_autoupdate
+    mode http
+    timeout server 1h
+    server autoupdate autoupdate:8002 resolvers docker_resolver check ssl verify none alpn h2

haproxy/src/haproxy.dev.cfg

@@ -0,0 +1,20 @@
+frontend https
+    mode http
+    bind *:8000 ssl crt /usr/local/etc/haproxy/combined.pem alpn h2,http/1.1
+    default_backend backend_client
+
+    acl autoupdate path_beg -i /system
+    use_backend backend_autoupdate if autoupdate
+
+    acl server path_beg -i /apps /media/ /rest /server-version.txt
+    use_backend backend_server if server
+
+    stats enable
+    stats uri /stats
+    stats refresh 10s
+    stats auth admin:admin
+
+backend backend_client
+    mode http
+    timeout tunnel 1h
+    server client client:4200 resolvers docker_resolver no-check

haproxy/src/haproxy.prod.cfg

@@ -0,0 +1,26 @@
+frontend https
+    mode http
+    bind *:8000 ssl crt /usr/local/etc/haproxy/combined.pem alpn h2,http/1.1
+    default_backend backend_client
+
+    acl autoupdate path_beg -i /system
+    use_backend backend_autoupdate if autoupdate
+
+    acl server path_beg -i /apps /rest /server-version.txt
+    use_backend backend_server if server
+
+    acl media path_beg -i /media/
+    use_backend backend_media if media
+
+    stats enable
+    stats uri /stats
+    stats refresh 10s
+    stats auth admin:admin
+
+backend backend_client
+    mode http
+    server client client:80 resolvers docker_resolver check
+
+backend backend_media
+    mode http
+    server media media:8000 resolvers docker_resolver check

server/.dockerignore

@@ -3,3 +3,6 @@
 **/.venv
 tests/
 personal_data/
+**/.mypy_cache
+**/.pytest_cache
+**/tests/file

server/docker/Dockerfile

@@ -70,5 +70,4 @@ COPY manage.py /app/
 COPY openslides /app/openslides
 COPY docker/server-version.txt /app/openslides/core/static/server-version.txt
 ENTRYPOINT ["/usr/local/sbin/entrypoint"]
-CMD ["gunicorn", "-w", "8", "--preload", "-b", "0.0.0.0:8000", "-k", \
+CMD ["gunicorn", "-w", "8", "--preload", "-t", "240", "-b", "0.0.0.0:8000", "openslides.wsgi"]
-  "uvicorn.workers.UvicornWorker", "openslides.asgi:application"]

server/docker/Dockerfile.dev

@@ -0,0 +1,30 @@
+FROM python:3.7-slim AS base
+
+# Variables relevant for CMD
+ENV DJANGO_SETTINGS_MODULE settings
+ENV PYTHONPATH personal_data/var/
+
+WORKDIR /app
+RUN apt-get -y update &&  apt-get install --no-install-recommends -y \
+    postgresql-client \
+    wait-for-it \
+    gcc \
+    git \
+    libxml2-dev \
+    libxmlsec1-dev \
+    libxmlsec1-openssl \
+    pkg-config
+  
+RUN rm -rf /var/lib/apt/lists/*
+
+COPY requirements /app/requirements
+COPY requirements.txt /app/requirements.txt
+
+RUN pip install -r requirements.txt -r requirements/saml.txt && \
+    rm -rf /root/.cache/pip
+
+EXPOSE 8000
+COPY docker/entrypoint-dev /usr/local/sbin/
+COPY . .
+ENTRYPOINT ["/usr/local/sbin/entrypoint-dev"]
+CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

server/docker/entrypoint-db-setup

@@ -37,7 +37,6 @@ done
 # Wait for redis
 wait-for-it redis:6379
 wait-for-it redis-slave:6379
-wait-for-it redis-channels:6379
 
 echo 'running migrations'
 python manage.py migrate

server/docker/entrypoint-dev

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+set -e
+
+wait-for-it -t 0 redis:6379
+
+if [[ ! -f "/app/personal_data/var/settings.py" ]]; then
+    echo "Create settings"
+    python manage.py createsettings
+fi
+
+python -u manage.py migrate
+
+exec "$@"

server/docker/settings.py

@@ -54,7 +54,7 @@ DEBUG = False
 RESET_PASSWORD_VERBOSE_ERRORS = get_env("RESET_PASSWORD_VERBOSE_ERRORS", True, bool)
 
 # OpenSlides specific settings
-AUTOUPDATE_DELAY = get_env("AUTOUPDATE_DELAY", 1, int)
+AUTOUPDATE_DELAY = get_env("AUTOUPDATE_DELAY", 1, float)
 DEMO_USERS = get_env("DEMO_USERS", default=None)
 DEMO_USERS = json.loads(DEMO_USERS) if DEMO_USERS else None
 
@@ -99,25 +99,10 @@ REDIS_HOST = get_env("REDIS_HOST", "redis")
 REDIS_PORT = get_env("REDIS_PORT", 6379, int)
 REDIS_SLAVE_HOST = get_env("REDIS_SLAVE_HOST", "redis-slave")
 REDIS_SLAVE_PORT = get_env("REDIS_SLAVE_PORT", 6379, int)
-REDIS_CHANNLES_HOST = get_env("REDIS_CHANNLES_HOST", "redis-channels")
-REDIS_CHANNLES_PORT = get_env("REDIS_CHANNLES_PORT", 6379, int)
-REDIS_SLAVE_WAIT_TIMEOUT = get_env("REDIS_SLAVE_WAIT_TIMEOUT", 10000, int)
-
-# Django Channels
-CHANNEL_LAYERS = {
-    "default": {
-        "BACKEND": "channels_redis.core.RedisChannelLayer",
-        "CONFIG": {
-            "hosts": [(REDIS_CHANNLES_HOST, REDIS_CHANNLES_PORT)],
-            "capacity": 10000,
-        },
-    },
-}
 
 # Collection Cache
 REDIS_ADDRESS = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
 REDIS_READ_ONLY_ADDRESS = f"redis://{REDIS_SLAVE_HOST}:{REDIS_SLAVE_PORT}/0"
-AMOUNT_REPLICAS = get_env("AMOUNT_REPLICAS", 1, int)
 CONNECTION_POOL_LIMIT = get_env("CONNECTION_POOL_LIMIT", 100, int)
 
 # Session backend
@@ -135,8 +120,6 @@ ENABLE_SAML = get_env("ENABLE_SAML", False, bool)
 if ENABLE_SAML:
     INSTALLED_APPS += ["openslides.saml"]
 
-# TODO: More saml stuff...
-
 # Controls if electronic voting (means non-analog polls) are enabled.
 ENABLE_ELECTRONIC_VOTING = get_env("ENABLE_ELECTRONIC_VOTING", False, bool)
 

server/openslides/agenda/apps.py

@@ -15,7 +15,6 @@ class AgendaAppConfig(AppConfig):
         from ..utils.access_permissions import required_user
         from ..utils.rest_api import router
         from . import serializers  # noqa
-        from .projector import register_projector_slides
         from .signals import (
             get_permission_change_data,
             listen_to_related_object_post_delete,
@@ -23,9 +22,6 @@ class AgendaAppConfig(AppConfig):
         )
         from .views import ItemViewSet, ListOfSpeakersViewSet
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         post_save.connect(
             listen_to_related_object_post_save,

server/openslides/agenda/projector.py

@@ -1,309 +0,0 @@
-from collections import defaultdict
-from typing import Any, Dict, List, Union
-
-from ..users.projector import get_user_name
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    ProjectorElementException,
-    get_config,
-    get_model,
-    register_projector_slide,
-)
-
-
-# Important: All functions have to be prune. This means, that thay can only
-#            access the data, that they get as argument and do not have any
-#            side effects.
-
-
-async def get_sorted_agenda_items(
-    agenda_items: Dict[int, Dict[str, Any]]
-) -> List[Dict[str, Any]]:
-    """
-    Returns all sorted agenda items by id first and then weight, resulting in
-    ordered items, if some have the same weight.
-    """
-    return sorted(
-        sorted(agenda_items.values(), key=lambda item: item["id"]),
-        key=lambda item: item["weight"],
-    )
-
-
-async def get_flat_tree(
-    agenda_items: Dict[int, Dict[str, Any]], parent_id: int = 0
-) -> List[Dict[str, Any]]:
-    """
-    Build the item tree from all_data_provider.
-
-    Only build the tree from elements unterneath parent_id.
-
-    Returns a list of two element tuples where the first element is the item title
-    and the second a List with children as two element tuples.
-    """
-
-    # Build a dict from an item_id to all its children
-    children: Dict[int, List[int]] = defaultdict(list)
-
-    for item in await get_sorted_agenda_items(agenda_items):
-        if item["type"] == 1:  # only normal items
-            children[item["parent_id"] or 0].append(item["id"])
-
-    tree = []
-
-    def build_tree(item_ids: List[int], depth: int) -> None:
-        for item_id in item_ids:
-            item = agenda_items[item_id]
-            title_information = item["title_information"]
-            title_information["_agenda_item_number"] = item["item_number"]
-            tree.append(
-                {
-                    "title_information": title_information,
-                    "collection": item["content_object"]["collection"],
-                    "depth": depth,
-                }
-            )
-            build_tree(children[item_id], depth + 1)
-
-    build_tree(children[parent_id], 0)
-    return tree
-
-
-async def item_list_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Item list slide.
-
-    Returns all root items or all children of an item.
-    """
-    # fetch all items, so they are cached:
-    all_agenda_items = await all_data_provider.get_collection("agenda/item")
-
-    only_main_items = element.get("only_main_items", True)
-    if only_main_items:
-        agenda_items = []
-        for item in await get_sorted_agenda_items(all_agenda_items):
-            if item["parent_id"] is None and item["type"] == 1:
-                title_information = item["title_information"]
-                title_information["_agenda_item_number"] = item["item_number"]
-                agenda_items.append(
-                    {
-                        "title_information": title_information,
-                        "collection": item["content_object"]["collection"],
-                    }
-                )
-    else:
-        agenda_items = await get_flat_tree(all_agenda_items)
-
-    return {"items": agenda_items}
-
-
-async def list_of_speakers_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    List of speakers slide.
-
-    Returns all usernames, that are on the list of speaker of a slide.
-    """
-    list_of_speakers_id = element.get("id")
-
-    if list_of_speakers_id is None:
-        raise ProjectorElementException("id is required for list of speakers slide")
-
-    return await get_list_of_speakers_slide_data(all_data_provider, list_of_speakers_id)
-
-
-async def get_list_of_speakers_slide_data(
-    all_data_provider: ProjectorAllDataProvider, list_of_speakers_id: int
-) -> Dict[str, Any]:
-    list_of_speakers = await get_model(
-        all_data_provider, "agenda/list-of-speakers", list_of_speakers_id
-    )
-
-    title_information = list_of_speakers["title_information"]
-    # try to get the agenda item for the content object (which must not exist)
-    content_object = await get_model(
-        all_data_provider,
-        list_of_speakers["content_object"]["collection"],
-        list_of_speakers["content_object"]["id"],
-    )
-    agenda_item_id = content_object.get("agenda_item_id")
-    if agenda_item_id is not None:
-        agenda_item = await all_data_provider.get("agenda/item", agenda_item_id)
-        if agenda_item is not None:
-            title_information["_agenda_item_number"] = agenda_item["item_number"]
-
-    # Partition speaker objects to waiting, current and finished
-    speakers_waiting = []
-    speakers_finished = []
-    current_speaker = None
-    for speaker in list_of_speakers["speakers"]:
-        user = await get_user_name(all_data_provider, speaker["user_id"])
-        formatted_speaker = {
-            "user": user,
-            "marked": speaker["marked"],
-            "point_of_order": speaker["point_of_order"],
-            "weight": speaker["weight"],
-            "end_time": speaker["end_time"],
-        }
-
-        if speaker["begin_time"] is None and speaker["end_time"] is None:
-            speakers_waiting.append(formatted_speaker)
-        elif speaker["begin_time"] is not None and speaker["end_time"] is None:
-            current_speaker = formatted_speaker
-        else:
-            speakers_finished.append(formatted_speaker)
-
-    # sort speakers
-    speakers_waiting = sorted(speakers_waiting, key=lambda s: s["weight"])
-    speakers_finished = sorted(speakers_finished, key=lambda s: s["end_time"])
-
-    number_of_last_speakers = await get_config(
-        all_data_provider, "agenda_show_last_speakers"
-    )
-    number_of_next_speakers = await get_config(
-        all_data_provider, "agenda_show_next_speakers"
-    )
-
-    if number_of_last_speakers == 0:
-        speakers_finished = []
-    else:
-        # Take the last speakers
-        speakers_finished = speakers_finished[-number_of_last_speakers:]
-
-    if number_of_next_speakers != -1:
-        speakers_waiting = speakers_waiting[:number_of_next_speakers]
-
-    return {
-        "waiting": speakers_waiting,
-        "current": current_speaker,
-        "finished": speakers_finished,
-        "content_object_collection": list_of_speakers["content_object"]["collection"],
-        "title_information": title_information,
-        "closed": list_of_speakers["closed"],
-    }
-
-
-async def get_current_list_of_speakers_id_for_projector(
-    all_data_provider: ProjectorAllDataProvider, projector: Dict[str, Any]
-) -> Union[int, None]:
-    """
-    Search for elements, that do have a list of speakers:
-    Try to get a model by the collection and id in the element. This
-    model needs to have a 'list_of_speakers_id'. This list of speakers
-    must exist. The first matching element is taken.
-    """
-    elements = projector["elements"]
-    list_of_speakers_id = None
-    for element in elements:
-        if "id" not in element:
-            continue
-        collection = element["name"]
-        id = element["id"]
-        model = await all_data_provider.get(collection, id)
-        if model is None:
-            continue
-
-        if "list_of_speakers_id" not in model:
-            continue
-
-        list_of_speakers_id = model["list_of_speakers_id"]
-        los_exists = await all_data_provider.exists(
-            "agenda/list-of-speakers", list_of_speakers_id
-        )
-        if not los_exists:
-            continue
-
-        break
-
-    return list_of_speakers_id
-
-
-async def get_reference_projector(
-    all_data_provider: ProjectorAllDataProvider, projector_id: int
-) -> Dict[str, Any]:
-    """
-    Returns the reference projector to the given projector (by id)
-    """
-    this_projector = await get_model(all_data_provider, "core/projector", projector_id)
-
-    reference_projector_id = this_projector["reference_projector_id"] or projector_id
-    return await get_model(all_data_provider, "core/projector", reference_projector_id)
-
-
-async def current_list_of_speakers_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    The current list of speakers slide. Creates the data for the given projector.
-    """
-    reference_projector = await get_reference_projector(all_data_provider, projector_id)
-    list_of_speakers_id = await get_current_list_of_speakers_id_for_projector(
-        all_data_provider, reference_projector
-    )
-    if list_of_speakers_id is None:  # no element found
-        return {}
-
-    return await get_list_of_speakers_slide_data(all_data_provider, list_of_speakers_id)
-
-
-async def current_speaker_chyron_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Returns the username for the current speaker.
-    """
-    # get projector for color information
-    projector = await get_model(all_data_provider, "core/projector", projector_id)
-
-    slide_data = {
-        "background_color": projector["chyron_background_color"],
-        "font_color": projector["chyron_font_color"],
-    }
-
-    reference_projector = await get_reference_projector(all_data_provider, projector_id)
-    list_of_speakers_id = await get_current_list_of_speakers_id_for_projector(
-        all_data_provider, reference_projector
-    )
-    if list_of_speakers_id is None:  # no element found
-        return slide_data
-
-    # get list of speakers to search current speaker
-    list_of_speakers = await get_model(
-        all_data_provider, "agenda/list-of-speakers", list_of_speakers_id
-    )
-
-    # find current speaker
-    current_speaker = None
-    for speaker in list_of_speakers["speakers"]:
-        if speaker["begin_time"] is not None and speaker["end_time"] is None:
-            current_speaker = await get_user_name(all_data_provider, speaker["user_id"])
-            break
-
-    if current_speaker is not None:
-        slide_data["current_speaker"] = current_speaker
-
-    return slide_data
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("agenda/item-list", item_list_slide)
-    register_projector_slide("agenda/list-of-speakers", list_of_speakers_slide)
-    register_projector_slide(
-        "agenda/current-list-of-speakers", current_list_of_speakers_slide
-    )
-    register_projector_slide(
-        "agenda/current-list-of-speakers-overlay", current_list_of_speakers_slide
-    )
-    register_projector_slide(
-        "agenda/current-speaker-chyron", current_speaker_chyron_slide
-    )

server/openslides/assignments/apps.py

@@ -13,7 +13,6 @@ class AssignmentsAppConfig(AppConfig):
         from ..utils.access_permissions import required_user
         from ..utils.rest_api import router
         from . import serializers  # noqa
-        from .projector import register_projector_slides
         from .signals import get_permission_change_data
         from .views import (
             AssignmentOptionViewSet,
@@ -22,9 +21,6 @@ class AssignmentsAppConfig(AppConfig):
             AssignmentVoteViewSet,
         )
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         permission_change.connect(
             get_permission_change_data,

server/openslides/assignments/projector.py

@@ -1,113 +0,0 @@
-from typing import Any, Dict, List
-
-from ..users.projector import get_user_name
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    get_model,
-    get_models,
-    register_projector_slide,
-)
-from .models import AssignmentPoll
-
-
-async def assignment_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Assignment slide.
-    """
-    assignment = await get_model(
-        all_data_provider, "assignments/assignment", element.get("id")
-    )
-
-    assignment_related_users: List[Dict[str, Any]] = [
-        {"user": await get_user_name(all_data_provider, aru["user_id"])}
-        for aru in sorted(
-            assignment["assignment_related_users"], key=lambda aru: aru["weight"]
-        )
-    ]
-
-    return {
-        "title": assignment["title"],
-        "phase": assignment["phase"],
-        "open_posts": assignment["open_posts"],
-        "description": assignment["description"],
-        "assignment_related_users": assignment_related_users,
-        "number_poll_candidates": assignment["number_poll_candidates"],
-    }
-
-
-async def assignment_poll_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Poll slide.
-    """
-    poll = await get_model(
-        all_data_provider, "assignments/assignment-poll", element.get("id")
-    )
-    assignment = await get_model(
-        all_data_provider, "assignments/assignment", poll["assignment_id"]
-    )
-
-    poll_data = {
-        key: poll[key]
-        for key in (
-            "title",
-            "type",
-            "pollmethod",
-            "min_votes_amount",
-            "max_votes_amount",
-            "description",
-            "state",
-            "onehundred_percent_base",
-            "majority_method",
-        )
-    }
-
-    # Add options:
-    poll_data["options"] = []
-    options = await get_models(
-        all_data_provider, "assignments/assignment-option", poll["options_id"]
-    )
-    for option in sorted(options, key=lambda option: option["weight"]):
-        option_data: Dict[str, Any] = {
-            "user": {
-                "short_name": await get_user_name(all_data_provider, option["user_id"])
-            }
-        }
-        if poll["state"] == AssignmentPoll.STATE_PUBLISHED:
-            option_data["yes"] = float(option["yes"])
-            option_data["no"] = float(option["no"])
-            option_data["abstain"] = float(option["abstain"])
-        poll_data["options"].append(option_data)
-
-    if poll["state"] == AssignmentPoll.STATE_PUBLISHED:
-        poll_data["amount_global_yes"] = (
-            float(poll["amount_global_yes"]) if poll["amount_global_yes"] else None
-        )
-        poll_data["amount_global_no"] = (
-            float(poll["amount_global_no"]) if poll["amount_global_no"] else None
-        )
-        poll_data["amount_global_abstain"] = (
-            float(poll["amount_global_abstain"])
-            if poll["amount_global_abstain"]
-            else None
-        )
-        poll_data["votesvalid"] = float(poll["votesvalid"])
-        poll_data["votesinvalid"] = float(poll["votesinvalid"])
-        poll_data["votescast"] = float(poll["votescast"])
-
-    return {
-        "assignment": {"title": assignment["title"]},
-        "poll": poll_data,
-    }
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("assignments/assignment", assignment_slide)
-    register_projector_slide("assignments/assignment-poll", assignment_poll_slide)

server/openslides/core/apps.py

@@ -16,11 +16,9 @@ class CoreAppConfig(AppConfig):
     def ready(self):
         # Import all required stuff.
         # Let all client websocket message register
-        from ..utils import websocket_client_messages  # noqa
         from ..utils.rest_api import router
         from . import serializers  # noqa
         from .config import config
-        from .projector import register_projector_slides
         from .signals import (
             autoupdate_for_many_to_many_relations,
             cleanup_unused_permissions,
@@ -41,9 +39,6 @@ class CoreAppConfig(AppConfig):
         # Collect all config variables before getting the constants.
         config.collect_config_variables_from_apps()
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         post_permission_creation.connect(
             delete_django_app_permissions, dispatch_uid="delete_django_app_permissions"
@@ -126,6 +121,7 @@ class CoreAppConfig(AppConfig):
 
         # Client settings
         client_settings_keys = [
+            "AUTOUPDATE_DELAY",
             "PRIORITIZED_GROUP_IDS",
             "PING_INTERVAL",
             "PING_TIMEOUT",

server/openslides/core/projector.py

@@ -1,68 +0,0 @@
-from typing import Any, Dict
-
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    get_config,
-    get_model,
-    register_projector_slide,
-)
-
-
-async def countdown_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Countdown slide.
-
-    Returns the full_data of the countdown element.
-
-    element = {
-        name: 'core/countdown',
-        id: 5,  # Countdown ID
-    }
-    """
-    countdown = await get_model(all_data_provider, "core/countdown", element.get("id"))
-    return {
-        "description": countdown["description"],
-        "running": countdown["running"],
-        "countdown_time": countdown["countdown_time"],
-        "warning_time": await get_config(
-            all_data_provider, "agenda_countdown_warning_time"
-        ),
-    }
-
-
-async def message_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Message slide.
-
-    Returns the full_data of the message element.
-
-    element = {
-        name: 'core/projector-message',
-        id: 5,  # ProjectorMessage ID
-    }
-    """
-    return await get_model(
-        all_data_provider, "core/projector-message", element.get("id")
-    )
-
-
-async def clock_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    return {}
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("core/countdown", countdown_slide)
-    register_projector_slide("core/projector-message", message_slide)
-    register_projector_slide("core/clock", clock_slide)

server/openslides/core/serializers.py

@@ -1,7 +1,6 @@
 from typing import Any
 
 from ..core.config import config
-from ..utils.projector import projector_slides
 from ..utils.rest_api import (
     Field,
     IdPrimaryKeyRelatedField,
@@ -62,10 +61,6 @@ def elements_validator(value: Any) -> None:
             raise ValidationError(
                 {"detail": "Every dictionary must have a key 'name'."}
             )
-        if element["name"] not in projector_slides:
-            raise ValidationError(
-                {"detail": "Unknown projector element {0}.", "args": [element["name"]]}
-            )
 
 
 def elements_array_validator(value: Any) -> None:

server/openslides/core/urls.py

@@ -4,7 +4,8 @@ from . import views
 
 
 urlpatterns = [
-    url(r"^servertime/$", views.ServerTime.as_view(), name="core_servertime"),
+    url(r"^servertime/$", views.ServertimeView.as_view(), name="core_servertime"),
+    url(r"^constants/$", views.ConstantsView.as_view(), name="core_constants"),
     url(r"^version/$", views.VersionView.as_view(), name="core_version"),
     url(
         r"^history/information/$",

server/openslides/core/views.py

@@ -22,6 +22,7 @@ from ..utils.arguments import arguments
 from ..utils.auth import GROUP_ADMIN_PK, anonymous_is_enabled, has_perm, in_some_groups
 from ..utils.autoupdate import inform_changed_data
 from ..utils.cache import element_cache
+from ..utils.constants import get_constants
 from ..utils.plugins import (
     get_plugin_description,
     get_plugin_license,
@@ -569,7 +570,7 @@ class CountdownViewSet(ModelViewSet):
 # Special API views
 
 
-class ServerTime(utils_views.APIView):
+class ServertimeView(utils_views.APIView):
     """
     Returns the server time as UNIX timestamp.
     """
@@ -580,6 +581,19 @@ class ServerTime(utils_views.APIView):
         return now().timestamp()
 
 
+class ConstantsView(utils_views.APIView):
+    """
+    Returns the server time as UNIX timestamp.
+    """
+
+    http_method_names = ["get"]
+
+    def get_context_data(self, **context):
+        if not self.request.user.is_authenticated and not anonymous_is_enabled():
+            self.permission_denied(self.request)
+        return get_constants()
+
+
 class VersionView(utils_views.APIView):
     """
     Returns a dictionary with the OpenSlides version and the version of all

server/openslides/global_settings.py

@@ -16,7 +16,6 @@ INSTALLED_APPS = [
     "django.contrib.sessions",
     "django.contrib.staticfiles",
     "rest_framework",
-    "channels",
     "openslides.agenda",
     "openslides.topics",
     "openslides.motions",
@@ -122,13 +121,5 @@ PASSWORD_HASHERS = [
 MEDIA_URL = "/media/"
 
 
-# Django Channels
-# http://channels.readthedocs.io/en/latest/
-
-ASGI_APPLICATION = "openslides.routing.application"
-
-CHANNEL_LAYERS = {"default": {"BACKEND": "channels.layers.InMemoryChannelLayer"}}
-
-
 # Enable updating the last_login field for users on every login.
 ENABLE_LAST_LOGIN_FIELD = False

server/openslides/mediafiles/apps.py

@@ -11,9 +11,7 @@ class MediafilesAppConfig(AppConfig):
         # Import all required stuff.
         from openslides.core.signals import permission_change
         from openslides.utils.rest_api import router
-
         from . import serializers  # noqa
-        from .projector import register_projector_slides
         from .signals import get_permission_change_data
         from .views import MediafileViewSet
 
@@ -25,9 +23,6 @@ class MediafilesAppConfig(AppConfig):
                 "The MEDIA_URL setting must start and end with a slash"
             )
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         permission_change.connect(
             get_permission_change_data,

server/openslides/mediafiles/projector.py

@@ -1,29 +0,0 @@
-from typing import Any, Dict
-
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    get_model,
-    register_projector_slide,
-)
-
-
-async def mediafile_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Slide for Mediafile.
-    """
-    mediafile = await get_model(
-        all_data_provider, "mediafiles/mediafile", element.get("id")
-    )
-    return {
-        "path": mediafile["path"],
-        "mimetype": mediafile["mimetype"],
-        "media_url_prefix": mediafile["media_url_prefix"],
-    }
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("mediafiles/mediafile", mediafile_slide)

server/openslides/motions/apps.py

@@ -12,10 +12,8 @@ class MotionsAppConfig(AppConfig):
         # Import all required stuff.
         from openslides.core.signals import permission_change
         from openslides.utils.rest_api import router
-
         from ..utils.access_permissions import required_user
         from . import serializers  # noqa
-        from .projector import register_projector_slides
         from .signals import create_builtin_workflows, get_permission_change_data
         from .views import (
             CategoryViewSet,
@@ -31,9 +29,6 @@ class MotionsAppConfig(AppConfig):
             WorkflowViewSet,
         )
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         post_migrate.connect(
             create_builtin_workflows, dispatch_uid="motion_create_builtin_workflows"

server/openslides/motions/projector.py

@@ -1,437 +0,0 @@
-import re
-from typing import Any, Dict, List, Optional
-
-from ..users.projector import get_user_name
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    ProjectorElementException,
-    get_config,
-    get_model,
-    register_projector_slide,
-)
-from .models import MotionPoll
-
-
-motion_placeholder_regex = re.compile(r"\[motion:(\d+)\]")
-
-
-async def get_state(
-    all_data_provider: ProjectorAllDataProvider,
-    motion: Dict[str, Any],
-    state_id_key: str,
-) -> Dict[str, Any]:
-    """
-    Returns a state element from one motion. Raises an error if the state does not exist.
-    """
-    state = await all_data_provider.get("motions/state", motion[state_id_key])
-    if state is None:
-        raise ProjectorElementException(
-            f"motion {motion['id']} can not be on the state with id {motion[state_id_key]}"
-        )
-    return state
-
-
-async def get_amendment_merge_into_motion_diff(all_data_provider, amendment):
-    """
-    HINT: This implementation should be consistent to showInDiffView() in ViewMotionAmendedParagraph.ts
-    """
-    if amendment["state_id"] is None:
-        return 0
-
-    state = await get_state(all_data_provider, amendment, "state_id")
-    if state["merge_amendment_into_final"] == -1:
-        return 0
-    if state["merge_amendment_into_final"] == 1:
-        return 1
-
-    if amendment["recommendation_id"] is None:
-        return 0
-    recommendation = await get_state(all_data_provider, amendment, "recommendation_id")
-    if recommendation["merge_amendment_into_final"] == 1:
-        return 1
-
-    return 0
-
-
-async def get_amendment_merge_into_motion_final(all_data_provider, amendment):
-    """
-    HINT: This implementation should be consistent to showInFinalView() in ViewMotionAmendedParagraph.ts
-    """
-    if amendment["state_id"] is None:
-        return 0
-
-    state = await get_state(all_data_provider, amendment, "state_id")
-    if state["merge_amendment_into_final"] == 1:
-        return 1
-
-    return 0
-
-
-async def get_amendments_for_motion(motion, all_data_provider):
-    amendment_data = []
-    for amendment_id in motion["amendments_id"]:
-        amendment = await all_data_provider.get("motions/motion", amendment_id)
-
-        merge_amendment_into_final = await get_amendment_merge_into_motion_final(
-            all_data_provider, amendment
-        )
-        merge_amendment_into_diff = await get_amendment_merge_into_motion_diff(
-            all_data_provider, amendment
-        )
-
-        # Add change recommendations to the amendments:
-        change_recommendations = []  # type: ignore
-        for change_recommendation_id in amendment["change_recommendations_id"]:
-            cr = await get_model(
-                all_data_provider,
-                "motions/motion-change-recommendation",
-                change_recommendation_id,
-            )
-            if cr is not None and not cr["internal"] and not cr["rejected"]:
-                change_recommendations.append(cr)
-
-        amendment_data.append(
-            {
-                "id": amendment["id"],
-                "identifier": amendment["identifier"],
-                "title": amendment["title"],
-                "amendment_paragraphs": amendment["amendment_paragraphs"],
-                "change_recommendations": change_recommendations,
-                "merge_amendment_into_diff": merge_amendment_into_diff,
-                "merge_amendment_into_final": merge_amendment_into_final,
-            }
-        )
-    return amendment_data
-
-
-async def get_amendment_base_motion(amendment, all_data_provider):
-    motion = await get_model(
-        all_data_provider, "motions/motion", amendment.get("parent_id")
-    )
-
-    return {
-        "identifier": motion["identifier"],
-        "title": motion["title"],
-        "text": motion["text"],
-    }
-
-
-async def get_amendment_base_statute(amendment, all_data_provider):
-    statute = await get_model(
-        all_data_provider,
-        "motions/statute-paragraph",
-        amendment.get("statute_paragraph_id"),
-    )
-    return {"title": statute["title"], "text": statute["text"]}
-
-
-async def extend_reference_motion_dict(
-    all_data_provider: ProjectorAllDataProvider,
-    recommendation: Optional[str],
-    referenced_motions: Dict[int, Dict[str, str]],
-) -> None:
-    """
-    Extends a dict of motion ids mapped to their title information.
-    The client can replace the placeholders in the recommendation correctly.
-    """
-    if recommendation is None:
-        return
-
-    # Collect all meantioned motions via [motion:<id>]
-    referenced_ids = [
-        int(id) for id in motion_placeholder_regex.findall(recommendation)
-    ]
-    for id in referenced_ids:
-        # Put every referenced motion into the referenced_motions dict
-        referenced_motion = await all_data_provider.get("motions/motion", id)
-        if id not in referenced_motions and referenced_motion is not None:
-            referenced_motions[id] = {
-                "title": referenced_motion["title"],
-                "identifier": referenced_motion["identifier"],
-            }
-
-
-async def motion_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Motion slide.
-
-    The returned dict can contain the following fields:
-    * identifier
-    * title
-    * text
-    * submitters
-    * amendment_paragraphs
-    * is_child
-    * show_meta_box
-    * show_referring_motions
-    * reason
-    * modified_final_version
-    * recommendation
-    * recommendation_extension
-    * recommender
-    * change_recommendations
-    """
-    # Get motion
-    mode = element.get(
-        "mode", await get_config(all_data_provider, "motions_recommendation_text_mode")
-    )
-
-    # populate cache:
-
-    motion = await get_model(all_data_provider, "motions/motion", element.get("id"))
-
-    # Add submitters
-    submitters = [
-        await get_user_name(all_data_provider, submitter["user_id"])
-        for submitter in sorted(
-            motion["submitters"], key=lambda submitter: submitter["weight"]
-        )
-    ]
-
-    # Get some needed config values
-    show_meta_box = not await get_config(
-        all_data_provider, "motions_disable_sidebox_on_projector"
-    )
-    show_referring_motions = not await get_config(
-        all_data_provider, "motions_hide_referring_motions"
-    )
-    line_length = await get_config(all_data_provider, "motions_line_length")
-    line_numbering_mode = await get_config(
-        all_data_provider, "motions_default_line_numbering"
-    )
-    motions_preamble = await get_config(all_data_provider, "motions_preamble")
-
-    # Query all change-recommendation and amendment related things.
-    amendments = []  # type: ignore
-    base_motion = None
-    base_statute = None
-    if motion["statute_paragraph_id"]:
-        base_statute = await get_amendment_base_statute(motion, all_data_provider)
-    elif motion["parent_id"] is not None and motion["amendment_paragraphs"]:
-        base_motion = await get_amendment_base_motion(motion, all_data_provider)
-    else:
-        amendments = await get_amendments_for_motion(motion, all_data_provider)
-
-    change_recommendations = []  # type: ignore
-    for change_recommendation_id in motion["change_recommendations_id"]:
-        cr = await get_model(
-            all_data_provider,
-            "motions/motion-change-recommendation",
-            change_recommendation_id,
-        )
-        if cr is not None and not cr["internal"]:
-            change_recommendations.append(cr)
-
-    # The base return value. More fields will get added below.
-    return_value = {
-        "identifier": motion["identifier"],
-        "title": motion["title"],
-        "submitters": submitters,
-        "preamble": motions_preamble,
-        "amendment_paragraphs": motion["amendment_paragraphs"],
-        "base_motion": base_motion,
-        "base_statute": base_statute,
-        "is_child": bool(motion["parent_id"]),
-        "show_meta_box": show_meta_box,
-        "show_referring_motions": show_referring_motions,
-        "change_recommendations": change_recommendations,
-        "amendments": amendments,
-        "line_length": line_length,
-        "line_numbering_mode": line_numbering_mode,
-    }
-
-    if not await get_config(all_data_provider, "motions_disable_text_on_projector"):
-        return_value["text"] = motion["text"]
-
-    if not await get_config(all_data_provider, "motions_disable_reason_on_projector"):
-        return_value["reason"] = motion["reason"]
-
-    if mode == "final":
-        return_value["modified_final_version"] = motion["modified_final_version"]
-
-    # Add recommendation, if enabled in config (and the motion has one)
-    if (
-        not await get_config(
-            all_data_provider, "motions_disable_recommendation_on_projector"
-        )
-        and motion["recommendation_id"]
-    ):
-        recommendation_state = await get_state(
-            all_data_provider, motion, "recommendation_id"
-        )
-        return_value["recommendation"] = recommendation_state["recommendation_label"]
-        if recommendation_state["show_recommendation_extension_field"]:
-            recommendation_extension = motion["recommendation_extension"]
-            # All title information for referenced motions in the recommendation
-            referenced_motions: Dict[int, Dict[str, str]] = {}
-            await extend_reference_motion_dict(
-                all_data_provider, recommendation_extension, referenced_motions
-            )
-            return_value["recommendation_extension"] = recommendation_extension
-            return_value["referenced_motions"] = referenced_motions
-        if motion["statute_paragraph_id"]:
-            return_value["recommender"] = await get_config(
-                all_data_provider, "motions_statute_recommendations_by"
-            )
-        else:
-            return_value["recommender"] = await get_config(
-                all_data_provider, "motions_recommendations_by"
-            )
-
-    if show_referring_motions:
-        # Add recommendation-referencing motions
-        return_value[
-            "recommendation_referencing_motions"
-        ] = await get_recommendation_referencing_motions(
-            all_data_provider, motion["id"]
-        )
-
-    return return_value
-
-
-async def get_recommendation_referencing_motions(
-    all_data_provider: ProjectorAllDataProvider, motion_id: int
-) -> Optional[List[Dict[str, Any]]]:
-    """
-    Returns all title information for motions, that are referencing
-    the given motion (by id) in their recommendation. If there are no
-    motions, None is returned (instead of []).
-    """
-    recommendation_referencing_motions = []
-    all_motions = await all_data_provider.get_collection("motions/motion")
-    for motion in all_motions.values():
-        # Motion must have a recommendation and a recommendaiton extension
-        if not motion["recommendation_id"] or not motion["recommendation_extension"]:
-            continue
-
-        # The recommendation must allow the extension field (there might be left-overs
-        # in a motions recommendation extension..)
-        recommendation = await get_state(all_data_provider, motion, "recommendation_id")
-        if not recommendation["show_recommendation_extension_field"]:
-            continue
-
-        # Find referenced motion ids
-        referenced_ids = [
-            int(id)
-            for id in motion_placeholder_regex.findall(
-                motion["recommendation_extension"]
-            )
-        ]
-
-        # if one of the referenced ids is the given motion, add the current motion.
-        if motion_id in referenced_ids:
-            recommendation_referencing_motions.append(
-                {"title": motion["title"], "identifier": motion["identifier"]}
-            )
-    return recommendation_referencing_motions or None
-
-
-async def motion_block_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Motion block slide.
-    """
-    motion_block = await get_model(
-        all_data_provider, "motions/motion-block", element.get("id")
-    )
-
-    # All motions in this motion block
-    motions = []
-
-    # All title information for referenced motions in the recommendation
-    referenced_motions: Dict[int, Dict[str, str]] = {}
-
-    # iterate motions.
-    for motion_id in motion_block["motions_id"]:
-        motion = await all_data_provider.get("motions/motion", motion_id)
-        # primarily to please mypy, should theoretically not happen
-        if motion is None:
-            raise RuntimeError(
-                f"motion {motion_id} of block {element.get('id')} could not be found"
-            )
-
-        motion_object = {
-            "title": motion["title"],
-            "identifier": motion["identifier"],
-        }
-
-        recommendation_id = motion["recommendation_id"]
-        if recommendation_id is not None:
-            recommendation = await get_state(
-                all_data_provider, motion, "recommendation_id"
-            )
-            motion_object["recommendation"] = {
-                "name": recommendation["recommendation_label"],
-                "css_class": recommendation["css_class"],
-            }
-            if recommendation["show_recommendation_extension_field"]:
-                recommendation_extension = motion["recommendation_extension"]
-                await extend_reference_motion_dict(
-                    all_data_provider, recommendation_extension, referenced_motions
-                )
-                motion_object["recommendation_extension"] = recommendation_extension
-
-        motions.append(motion_object)
-
-    return {
-        "title": motion_block["title"],
-        "motions": motions,
-        "referenced_motions": referenced_motions,
-    }
-
-
-async def motion_poll_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Poll slide.
-    """
-    poll = await get_model(all_data_provider, "motions/motion-poll", element.get("id"))
-    motion = await get_model(all_data_provider, "motions/motion", poll["motion_id"])
-
-    poll_data = {
-        key: poll[key]
-        for key in (
-            "title",
-            "type",
-            "pollmethod",
-            "state",
-            "onehundred_percent_base",
-            "majority_method",
-        )
-    }
-
-    if poll["state"] == MotionPoll.STATE_PUBLISHED:
-        option = await get_model(
-            all_data_provider, "motions/motion-option", poll["options_id"][0]
-        )  # there can only be exactly one option
-        poll_data["options"] = [
-            {
-                "yes": float(option["yes"]),
-                "no": float(option["no"]),
-                "abstain": float(option["abstain"]),
-            }
-        ]
-        poll_data["votesvalid"] = poll["votesvalid"]
-        poll_data["votesinvalid"] = poll["votesinvalid"]
-        poll_data["votescast"] = poll["votescast"]
-
-    return {
-        "motion": {"title": motion["title"], "identifier": motion["identifier"]},
-        "poll": poll_data,
-    }
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("motions/motion", motion_slide)
-    register_projector_slide("motions/motion-block", motion_block_slide)
-    register_projector_slide("motions/motion-poll", motion_poll_slide)

server/openslides/routing.py

@@ -1,15 +0,0 @@
-from channels.routing import ProtocolTypeRouter, URLRouter
-from django.conf.urls import url
-
-from openslides.utils.consumers import CloseConsumer, SiteConsumer
-from openslides.utils.middleware import AuthMiddlewareStack
-
-
-application = ProtocolTypeRouter(
-    {
-        # WebSocket chat handler
-        "websocket": AuthMiddlewareStack(
-            URLRouter([url(r"^ws/$", SiteConsumer), url(".*", CloseConsumer)])
-        )
-    }
-)

server/openslides/topics/apps.py

@@ -11,13 +11,9 @@ class TopicsAppConfig(AppConfig):
 
         from ..utils.rest_api import router
         from . import serializers  # noqa
-        from .projector import register_projector_slides
         from .signals import get_permission_change_data
         from .views import TopicViewSet
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         permission_change.connect(
             get_permission_change_data, dispatch_uid="topics_get_permission_change_data"

server/openslides/topics/projector.py

@@ -1,32 +0,0 @@
-from typing import Any, Dict
-
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    get_model,
-    register_projector_slide,
-)
-
-
-async def topic_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    Topic slide.
-
-    The returned dict can contain the following fields:
-    * title
-    * text
-    """
-    topic = await get_model(all_data_provider, "topics/topic", element.get("id"))
-    item = await get_model(all_data_provider, "agenda/item", topic["agenda_item_id"])
-    return {
-        "title": topic["title"],
-        "text": topic["text"],
-        "item_number": item["item_number"],
-    }
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("topics/topic", topic_slide)

server/openslides/users/apps.py

@@ -15,13 +15,9 @@ class UsersAppConfig(AppConfig):
         from ..core.signals import permission_change, post_permission_creation
         from ..utils.rest_api import router
         from . import serializers  # noqa
-        from .projector import register_projector_slides
         from .signals import create_builtin_groups_and_admin, get_permission_change_data
         from .views import GroupViewSet, PersonalNoteViewSet, UserViewSet
 
-        # Define projector elements.
-        register_projector_slides()
-
         # Connect signals.
         post_permission_creation.connect(
             create_builtin_groups_and_admin,

server/openslides/users/projector.py

@@ -1,44 +0,0 @@
-from typing import Any, Dict, List, Optional
-
-from ..utils.projector import (
-    ProjectorAllDataProvider,
-    get_model,
-    register_projector_slide,
-)
-
-
-async def user_slide(
-    all_data_provider: ProjectorAllDataProvider,
-    element: Dict[str, Any],
-    projector_id: int,
-) -> Dict[str, Any]:
-    """
-    User slide.
-
-    The returned dict can contain the following fields:
-    * user
-    """
-    return {"user": await get_user_name(all_data_provider, element.get("id"))}
-
-
-async def get_user_name(
-    all_data_provider: ProjectorAllDataProvider, user_id: Optional[int]
-) -> str:
-    """
-    Returns the short name for an user_id.
-    """
-    user = await get_model(all_data_provider, "users/user", user_id)
-
-    name_parts: List[str] = []
-    for name_part in ("title", "first_name", "last_name"):
-        if user[name_part]:
-            name_parts.append(user[name_part])
-    if not name_parts:
-        name_parts.append(user["username"])
-    if user["structure_level"]:
-        name_parts.append(f"({user['structure_level']})")
-    return " ".join(name_parts)
-
-
-def register_projector_slides() -> None:
-    register_projector_slide("users/user", user_slide)

server/openslides/utils/autoupdate.py

@@ -4,13 +4,12 @@ from collections import defaultdict
 from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
 
 from asgiref.sync import async_to_sync
-from channels.layers import get_channel_layer
 from django.db.models import Model
 from mypy_extensions import TypedDict
 
 from .auth import UserDoesNotExist
 from .cache import ChangeIdTooLowError, element_cache, get_element_id
-from .projector import get_projector_data
+from .stream import stream
 from .timing import Timing
 from .utils import get_model_from_collection_string, is_iterable, split_element_id
 
@@ -112,8 +111,7 @@ class AutoupdateBundle:
         save_history(self.element_iterator)
 
         # Update cache and send autoupdate using async code.
-        change_id = async_to_sync(self.dispatch_autoupdate)()
+        return async_to_sync(self.dispatch_autoupdate)()
-        return change_id
 
     @property
     def element_iterator(self) -> Iterable[AutoupdateElement]:
@@ -121,7 +119,7 @@ class AutoupdateBundle:
         for elements in self.autoupdate_elements.values():
             yield from elements.values()
 
-    async def update_cache(self) -> int:
+    async def get_data_for_cache(self) -> Dict[str, Optional[Dict[str, Any]]]:
         """
         Async helper function to update the cache.
 
@@ -136,7 +134,7 @@ class AutoupdateBundle:
                     "no_delete_on_restriction", False
                 )
             cache_elements[element_id] = full_data
-        return await element_cache.change_elements(cache_elements)
+        return cache_elements
 
     async def dispatch_autoupdate(self) -> int:
         """
@@ -145,25 +143,12 @@ class AutoupdateBundle:
         Return the change_id
         """
         # Update cache
-        change_id = await self.update_cache()
+        cache_elements = await self.get_data_for_cache()
+        change_id = await element_cache.change_elements(cache_elements)
 
         # Send autoupdate
-        channel_layer = get_channel_layer()
+        autoupdate_payload = {"elements": cache_elements, "change_id": change_id}
-        await channel_layer.group_send(
+        await stream.send("autoupdate", autoupdate_payload)
-            "autoupdate", {"type": "msg_new_change_id", "change_id": change_id}
-        )
-
-        # Send projector
-        projector_data = await get_projector_data()
-        channel_layer = get_channel_layer()
-        await channel_layer.group_send(
-            "projector",
-            {
-                "type": "msg_projector_data",
-                "data": projector_data,
-                "change_id": change_id,
-            },
-        )
 
         return change_id
 
@@ -286,14 +271,12 @@ class AutoupdateBundleMiddleware:
         if status_ok or status_redirect:
             change_id = bundle.done()
 
-            # inject the autoupdate, if there is an autoupdate and the status is
+            # inject the change id, if there was an autoupdate and the response status is
             # ok (and not redirect; redirects do not have a useful content)
             if change_id is not None and status_ok:
-                user_id = request.user.pk or 0
                 # Inject the autoupdate in the response.
                 # The complete response body will be overwritten!
-                _, autoupdate = async_to_sync(get_autoupdate_data)(change_id, user_id)
+                content = {"change_id": change_id, "data": response.data}
-                content = {"autoupdate": autoupdate, "data": response.data}
                 # Note: autoupdate may be none on skipped ones (which should not happen
                 # since the user has made the request....)
                 response.content = json.dumps(content)

server/openslides/utils/cache.py

@@ -163,6 +163,8 @@ class ElementCache:
         logger.info("Saving cache data into the cache...")
         await self.cache_provider.add_to_full_data(mapping)
         logger.info("Done saving the cache data.")
+        await self.cache_provider.set_cache_ready()
+        logger.info("Done: Cache is ready now.")
 
     def _build_cache_get_elementid_model_mapping(
         self, config_only: bool = False

server/openslides/utils/cache_providers.py

@@ -8,11 +8,7 @@ from django.core.exceptions import ImproperlyConfigured
 from typing_extensions import Protocol
 
 from . import logging
-from .redis import (
+from .redis import use_redis
-    read_only_redis_amount_replicas,
-    read_only_redis_wait_timeout,
-    use_redis,
-)
 from .schema_version import SchemaVersion
 from .utils import split_element_id, str_dict_to_bytes
 
@@ -54,6 +50,9 @@ class ElementCacheProvider(Protocol):
     async def data_exists(self) -> bool:
         ...
 
+    async def set_cache_ready(self) -> None:
+        ...
+
     async def get_all_data(self) -> Dict[bytes, bytes]:
         ...
 
@@ -127,6 +126,7 @@ class RedisCacheProvider:
     full_data_cache_key: str = "full_data"
     change_id_cache_key: str = "change_id"
     schema_cache_key: str = "schema"
+    cache_ready_key: str = "cache_ready"
 
     # All lua-scripts used by this provider. Every entry is a Tuple (str, bool) with the
     # script and an ensure_cache-indicator. If the indicator is True, a short ensure_cache-script
@@ -325,6 +325,7 @@ class RedisCacheProvider:
         """
         async with get_connection() as redis:
             tr = redis.multi_exec()
+            tr.delete(self.cache_ready_key)
             tr.delete(self.change_id_cache_key)
             tr.delete(self.full_data_cache_key)
             tr.hmset_dict(self.full_data_cache_key, data)
@@ -342,11 +343,16 @@ class RedisCacheProvider:
         Returns True, when there is data in the cache.
         """
         async with get_connection(read_only=True) as redis:
-            return await redis.exists(self.full_data_cache_key) and bool(
+            return (await redis.get(self.cache_ready_key)) is not None
-                await redis.zrangebyscore(
+            # return await redis.exists(self.full_data_cache_key) and bool(
-                    self.change_id_cache_key, withscores=True, count=1, offset=0
+            #    await redis.zrangebyscore(
-                )
+            #        self.change_id_cache_key, withscores=True, count=1, offset=0
-            )
+            #    )
+            # )
+
+    async def set_cache_ready(self) -> None:
+        async with get_connection(read_only=False) as redis:
+            await redis.set(self.cache_ready_key, "ok")
 
     @ensure_cache_wrapper()
     async def get_all_data(self) -> Dict[bytes, bytes]:
@@ -495,7 +501,11 @@ class RedisCacheProvider:
     async def get_schema_version(self) -> Optional[SchemaVersion]:
         """ Retrieves the schema version of the cache or None, if not existent """
         async with get_connection(read_only=True) as redis:
-            schema_version = await redis.hgetall(self.schema_cache_key)
+            try:
+                schema_version = await redis.hgetall(self.schema_cache_key)
+            except aioredis.errors.ReplyError:
+                await redis.delete(self.schema_cache_key)
+                return None
         if not schema_version:
             return None
 
@@ -543,15 +553,6 @@ class RedisCacheProvider:
                     raise CacheReset()
                 else:
                     raise e
-            if not read_only and read_only_redis_amount_replicas is not None:
-                reported_amount = await redis.wait(
-                    read_only_redis_amount_replicas, read_only_redis_wait_timeout
-                )
-                if reported_amount != read_only_redis_amount_replicas:
-                    logger.warn(
-                        f"WAIT reported {reported_amount} replicas of {read_only_redis_amount_replicas} "
-                        + f"requested after {read_only_redis_wait_timeout} ms!"
-                    )
             return result
 
     async def _eval(
@@ -584,6 +585,7 @@ class MemoryCacheProvider:
         self.set_data_dicts()
 
     def set_data_dicts(self) -> None:
+        self.ready = False
         self.full_data: Dict[str, str] = {}
         self.change_id_data: Dict[int, Set[str]] = {}
         self.locks: Dict[str, str] = {}
@@ -594,6 +596,7 @@ class MemoryCacheProvider:
 
     async def clear_cache(self) -> None:
         self.set_data_dicts()
+        self.ready = False
 
     async def reset_full_cache(
         self, data: Dict[str, str], default_change_id: int
@@ -606,7 +609,11 @@ class MemoryCacheProvider:
         self.full_data.update(data)
 
     async def data_exists(self) -> bool:
-        return bool(self.full_data) and self.default_change_id >= 0
+        return self.ready
+        # return bool(self.full_data) and self.default_change_id >= 0
+
+    async def set_cache_ready(self) -> None:
+        self.ready = True
 
     async def get_all_data(self) -> Dict[bytes, bytes]:
         return str_dict_to_bytes(self.full_data)

server/openslides/utils/consumer_autoupdate_strategy.py

@@ -1,102 +0,0 @@
-import asyncio
-from asyncio import Task
-from typing import Optional, cast
-
-from django.conf import settings
-
-from .autoupdate import get_autoupdate_data
-from .cache import element_cache
-from .websocket import ChangeIdTooHighException, ProtocollAsyncJsonWebsocketConsumer
-
-
-AUTOUPDATE_DELAY = getattr(settings, "AUTOUPDATE_DELAY", None)
-
-
-class ConsumerAutoupdateStrategy:
-    def __init__(self, consumer: ProtocollAsyncJsonWebsocketConsumer) -> None:
-        self.consumer = consumer
-        # client_change_id = None: unknown -> set on first autoupdate or request_change_id
-        # client_change_id is int: the change_id, the client knows about, so the next
-        # update must be from client_change_id+1 .. <next clange_id>
-        self.client_change_id: Optional[int] = None
-        self.next_send_time = None
-        self.timer_task_handle: Optional[Task[None]] = None
-        self.lock = asyncio.Lock()
-
-    async def request_change_id(
-        self, change_id: int, in_response: Optional[str] = None
-    ) -> None:
-        """
-        The change id is not inclusive, so the client is on change_id and wants
-        data from change_id+1 .. now
-        """
-        # This resets the server side tracking of the client's change id.
-        async with self.lock:
-            await self.stop_timer()
-
-            max_change_id = await element_cache.get_current_change_id()
-            self.client_change_id = change_id
-
-            if self.client_change_id == max_change_id:
-                # The client is up-to-date, so nothing will be done
-                return None
-
-            if self.client_change_id > max_change_id:
-                message = (
-                    f"Requested change_id {self.client_change_id} is higher than the "
-                    + f"highest change_id {max_change_id}."
-                )
-                raise ChangeIdTooHighException(message, in_response=in_response)
-
-            await self.send_autoupdate(in_response=in_response)
-
-    async def new_change_id(self, change_id: int) -> None:
-        async with self.lock:
-            if self.client_change_id is None:
-                # The -1 is to send this autoupdate as the first one to he client.
-                # Remember: the client_change_id is the change_id the client knows about
-                self.client_change_id = change_id - 1
-
-            if AUTOUPDATE_DELAY is None:  # feature deactivated, send directly
-                await self.send_autoupdate()
-            elif self.timer_task_handle is None:
-                await self.start_timer()
-
-    async def get_running_loop(self) -> asyncio.AbstractEventLoop:
-        if hasattr(asyncio, "get_running_loop"):
-            return asyncio.get_running_loop()  # type: ignore
-        else:
-            return asyncio.get_event_loop()
-
-    async def start_timer(self) -> None:
-        loop = await self.get_running_loop()
-        self.timer_task_handle = loop.create_task(self.timer_task())
-
-    async def stop_timer(self) -> None:
-        if self.timer_task_handle is not None:
-            self.timer_task_handle.cancel()
-        self.timer_task_handle = None
-
-    async def timer_task(self) -> None:
-        try:
-            await asyncio.sleep(AUTOUPDATE_DELAY)
-        except asyncio.CancelledError:
-            return
-
-        async with self.lock:
-            await self.send_autoupdate()
-            self.timer_task_handle = None
-
-    async def send_autoupdate(self, in_response: Optional[str] = None) -> None:
-        # here, 1 is added to the change_id, because the client_change_id is the id the client
-        # *knows* about -> the client needs client_change_id+1 since get_autoupdate_data is
-        # inclusive [change_id .. max_change_id].
-        max_change_id, autoupdate = await get_autoupdate_data(
-            cast(int, self.client_change_id) + 1, self.consumer.user_id
-        )
-        if autoupdate is not None:
-            self.client_change_id = max_change_id
-            # It will be send, so we can set the client_change_id
-            await self.consumer.send_json(
-                type="autoupdate", content=autoupdate, in_response=in_response
-            )

server/openslides/utils/consumers.py

@@ -1,175 +0,0 @@
-import time
-from typing import Any, Dict, List, Optional, cast
-from urllib.parse import parse_qs
-
-from channels.generic.websocket import AsyncWebsocketConsumer
-
-from . import logging
-from .auth import UserDoesNotExist, async_anonymous_is_enabled
-from .cache import element_cache
-from .consumer_autoupdate_strategy import ConsumerAutoupdateStrategy
-from .utils import get_worker_id
-from .websocket import BaseWebsocketException, ProtocollAsyncJsonWebsocketConsumer
-
-
-logger = logging.getLogger("openslides.websocket")
-
-
-class SiteConsumer(ProtocollAsyncJsonWebsocketConsumer):
-    """
-    Websocket Consumer for the site.
-    """
-
-    groups = ["site"]
-
-    ID_COUNTER = 0
-    """
-    ID counter for assigning each instance of this class an unique id.
-    """
-
-    def __init__(self, *args: Any, **kwargs: Any) -> None:
-        self.projector_hash: Dict[int, int] = {}
-        SiteConsumer.ID_COUNTER += 1
-        self._id = get_worker_id() + "-" + str(SiteConsumer.ID_COUNTER)
-        self.autoupdate_strategy = ConsumerAutoupdateStrategy(self)
-        super().__init__(*args, **kwargs)
-
-    async def connect(self) -> None:
-        """
-        A user connects to the site.
-
-        If it is an anonymous user and anonymous is disabled, the connection is closed.
-
-        Sends the startup data to the user.
-        """
-        self.user_id = self.scope["user"]["id"]
-
-        self.connect_time = time.time()
-        # self.scope['user'] is the full_data dict of the user. For an
-        # anonymous user is it the dict {'id': 0}
-        change_id = None
-        if not await async_anonymous_is_enabled() and not self.user_id:
-            await self.accept()  # workaround for #4009
-            await self.close()
-            logger.debug(f"connect: denied ({self._id})")
-            return
-
-        query_string = cast(
-            Dict[bytes, List[bytes]], parse_qs(self.scope["query_string"])
-        )
-        if b"change_id" in query_string:
-            try:
-                change_id = int(query_string[b"change_id"][0])
-            except ValueError:
-                await self.accept()  # workaround for #4009
-                await self.close()
-                logger.debug(f"connect: wrong change id ({self._id})")
-                return
-
-        await self.accept()
-
-        if change_id is not None:
-            logger.debug(f"connect: change id {change_id} ({self._id})")
-            try:
-                await self.request_autoupdate(change_id)
-            except BaseWebsocketException as e:
-                await self.send_exception(e)
-        else:
-            logger.debug(f"connect: no change id ({self._id})")
-
-        await self.channel_layer.group_add("autoupdate", self.channel_name)
-
-    async def disconnect(self, close_code: int) -> None:
-        """
-        A user disconnects. Remove it from autoupdate.
-        """
-        await self.channel_layer.group_discard("autoupdate", self.channel_name)
-        active_seconds = int(time.time() - self.connect_time)
-        logger.debug(
-            f"disconnect code={close_code} active_secs={active_seconds} ({self._id})"
-        )
-
-    async def msg_new_change_id(self, event: Dict[str, Any]) -> None:
-        """
-        Send changed or deleted elements to the user.
-        """
-        change_id = event["change_id"]
-        try:
-            await self.autoupdate_strategy.new_change_id(change_id)
-        except UserDoesNotExist:
-            # Maybe the user was deleted, but a websocket connection is still open to the user.
-            # So we can close this connection and return.
-            await self.close()
-
-    async def msg_projector_data(self, event: Dict[str, Any]) -> None:
-        """
-        The projector has changed.
-        """
-        all_projector_data = event["data"]
-        change_id = event["change_id"]
-
-        projector_data: Dict[int, Dict[str, Any]] = {}
-        for projector_id in self.listen_projector_ids:
-            data = all_projector_data.get(projector_id, [])
-            new_hash = hash(str(data))
-            if new_hash != self.projector_hash.get(projector_id):
-                projector_data[projector_id] = data
-                self.projector_hash[projector_id] = new_hash
-
-        if projector_data:
-            await self.send_projector_data(projector_data, change_id=change_id)
-
-    async def msg_notify(self, event: Dict[str, Any]) -> None:
-        """
-        Send a notify message to the user.
-        """
-        item = event["incomming"]
-
-        users = item.get("users")
-        reply_channels = item.get("replyChannels")
-        if (
-            (isinstance(users, bool) and users)
-            or (isinstance(users, list) and self.user_id in users)
-            or (
-                isinstance(reply_channels, list) and self.channel_name in reply_channels
-            )
-            or (users is None and reply_channels is None)
-        ):
-            item["senderChannelName"] = event["senderChannelName"]
-            item["senderUserId"] = event["senderUserId"]
-            await self.send_json(type="notify", content=item)
-
-    async def request_autoupdate(
-        self, change_id: int, in_response: Optional[str] = None
-    ) -> None:
-        await self.autoupdate_strategy.request_change_id(
-            change_id, in_response=in_response
-        )
-
-    async def send_projector_data(
-        self,
-        data: Dict[int, Dict[str, Any]],
-        change_id: Optional[int] = None,
-        in_response: Optional[str] = None,
-    ) -> None:
-        """
-        Sends projector data to the consumer.
-        """
-        if change_id is None:
-            change_id = await element_cache.get_current_change_id()
-
-        content = {"change_id": change_id, "data": data}
-        await self.send_json(type="projector", content=content, in_response=in_response)
-
-
-class CloseConsumer(AsyncWebsocketConsumer):
-    """ Auto-closes the connection """
-
-    groups: List[str] = []
-
-    def __init__(self, args: Dict[str, Any], **kwargs: Any) -> None:
-        logger.info(f'Closing connection to unknown websocket url {args["path"]}')
-
-    async def connect(self) -> None:
-        await self.accept()
-        await self.close()

server/openslides/utils/middleware.py

@@ -1,73 +0,0 @@
-from typing import Any, Dict, Optional
-
-from channels.auth import (
-    AuthMiddleware,
-    CookieMiddleware,
-    SessionMiddleware,
-    _get_user_session_key,
-)
-from django.conf import settings
-from django.contrib.auth import BACKEND_SESSION_KEY, HASH_SESSION_KEY
-from django.utils.crypto import constant_time_compare
-
-from .cache import element_cache
-
-
-class CollectionAuthMiddleware(AuthMiddleware):
-    """
-    Like the channels AuthMiddleware but returns a user dict id instead of
-    a django Model as user.
-    """
-
-    def populate_scope(self, scope: Dict[str, Any]) -> None:
-        # Make sure we have a session
-        if "session" not in scope:
-            raise ValueError(
-                "AuthMiddleware cannot find session in scope. SessionMiddleware must be above it."
-            )
-        # Add it to the scope if it's not there already
-        if "user" not in scope:
-            scope["user"] = {}
-
-    async def resolve_scope(self, scope: Dict[str, Any]) -> None:
-        scope["user"].update(await get_user(scope))
-
-
-async def get_user(scope: Dict[str, Any]) -> Dict[str, Any]:
-    """
-    Returns a user id from a channels-scope-session.
-
-    If no user is retrieved, return {'id': 0}.
-    """
-    # This code is basicly from channels.auth:
-    # https://github.com/django/channels/blob/d5e81a78e96770127da79248349808b6ee6ec2a7/channels/auth.py#L16
-    if "session" not in scope:
-        raise ValueError(
-            "Cannot find session in scope. You should wrap your consumer in SessionMiddleware."
-        )
-    session = scope["session"]
-    user: Optional[Dict[str, Any]] = None
-    try:
-        user_id = _get_user_session_key(session)
-        backend_path = session[BACKEND_SESSION_KEY]
-    except KeyError:
-        pass
-    else:
-        if backend_path in settings.AUTHENTICATION_BACKENDS:
-            user = await element_cache.get_element_data("users/user", user_id)
-            if user:
-                # Verify the session
-                session_hash = session.get(HASH_SESSION_KEY)
-                session_hash_verified = session_hash and constant_time_compare(
-                    session_hash, user["session_auth_hash"]
-                )
-                if not session_hash_verified:
-                    session.flush()
-                    user = None
-    return user or {"id": 0}
-
-
-# Handy shortcut for applying all three layers at once
-AuthMiddlewareStack = lambda inner: CookieMiddleware(  # noqa
-    SessionMiddleware(CollectionAuthMiddleware(inner))
-)

server/openslides/utils/projector.py

@@ -1,172 +0,0 @@
-"""
-General projector code.
-
-Functions  that handel the registration of projector elements and the rendering
-of the data to present it on the projector.
-"""
-
-from collections import defaultdict
-from typing import Any, Awaitable, Callable, Dict, List, Optional
-
-from . import logging
-from .cache import element_cache
-
-
-logger = logging.getLogger(__name__)
-
-
-class ProjectorElementException(Exception):
-    """
-    Exception for errors in one element on the projector.
-    """
-
-
-class ProjectorAllDataProvider:
-    NON_EXISTENT_MARKER = object()
-
-    def __init__(self) -> None:
-        self.cache: Any = defaultdict(dict)  # fuu you mypy
-        self.fetched_collection: Dict[str, bool] = {}
-
-    async def get(self, collection: str, id: int) -> Optional[Dict[str, Any]]:
-        cache_data = self.cache[collection].get(id)
-        if cache_data is None:
-            data: Any = await element_cache.get_element_data(collection, id)
-            if data is None:
-                data = ProjectorAllDataProvider.NON_EXISTENT_MARKER
-            self.cache[collection][id] = data
-
-        cache_data = self.cache[collection][id]
-        if cache_data == ProjectorAllDataProvider.NON_EXISTENT_MARKER:
-            return None
-        return cache_data
-
-    async def get_collection(self, collection: str) -> Dict[int, Dict[str, Any]]:
-        if not self.fetched_collection.get(collection, False):
-            collection_data = await element_cache.get_collection_data(collection)
-            self.cache[collection] = collection_data
-            self.fetched_collection[collection] = True
-        return self.cache[collection]
-
-    async def exists(self, collection: str, id: int) -> bool:
-        model = await self.get(collection, id)
-        return model is not None
-
-
-ProjectorSlide = Callable[
-    [ProjectorAllDataProvider, Dict[str, Any], int], Awaitable[Dict[str, Any]]
-]
-
-
-projector_slides: Dict[str, ProjectorSlide] = {}
-
-
-def register_projector_slide(name: str, slide: ProjectorSlide) -> None:
-    """
-    Registers a projector slide.
-
-    Has to be called in the app.ready method.
-    """
-    projector_slides[name] = slide
-
-
-async def get_projector_data(
-    projector_ids: List[int] = None,
-) -> Dict[int, List[Dict[str, Any]]]:
-    """
-    Calculates and returns the data for one or all projectors.
-
-    The keys of the returned data are the projector ids as int. When converted
-    to json, the numbers will changed to strings like "1".
-
-    The data for each projector is a list of elements.
-
-    Each element is a dict where the keys are "elements", "data". "elements"
-    contains the projector elements. It is the same as the projector elements in
-    the database. "data" contains all necessary data to render the projector
-    element. The key can also be "error" if there is a generall error for the
-    slide. In this case the values "elements" and "data" are optional.
-
-    The returned value looks like this:
-
-    projector_data = {
-        1: [
-            {
-                "element": {
-                    "name": "agenda/item-list",
-                },
-                "data": {
-                    "items": []
-                },
-            },
-        ],
-    }
-    """
-    if projector_ids is None:
-        projector_ids = []
-
-    projector_data: Dict[int, List[Dict[str, Any]]] = {}
-    all_data_provider = ProjectorAllDataProvider()
-    projectors = await all_data_provider.get_collection("core/projector")
-
-    for projector_id, projector in projectors.items():
-        if projector_ids and projector_id not in projector_ids:
-            # only render the projector in question.
-            continue
-
-        if not projector["elements"]:
-            # Skip empty elements.
-            continue
-
-        projector_data[projector_id] = []
-        for element in projector["elements"]:
-            projector_slide = projector_slides[element["name"]]
-            try:
-                data = await projector_slide(all_data_provider, element, projector_id)
-            except ProjectorElementException as err:
-                data = {"error": str(err)}
-            projector_data[projector_id].append({"data": data, "element": element})
-
-    return projector_data
-
-
-async def get_config(all_data_provider: ProjectorAllDataProvider, key: str) -> Any:
-    """
-    Returns a config value from all_data_provider.
-    Triggers the cache early: It access `get_colelction` instead of `get`. It
-    allows for all successive queries for configs to be cached.
-    """
-    from ..core.config import config
-
-    config_id = (await config.async_get_key_to_id())[key]
-
-    configs = await all_data_provider.get_collection(config.get_collection_string())
-    return configs[config_id]["value"]
-
-
-async def get_model(
-    all_data_provider: ProjectorAllDataProvider, collection: str, id: Any
-) -> Dict[str, Any]:
-    """
-    Tries to get the model identified by the collection and id.
-    If the id is invalid or the model not found, ProjectorElementExceptions will be raised.
-    """
-    if id is None:
-        raise ProjectorElementException(f"id is required for {collection} slide")
-
-    model = await all_data_provider.get(collection, id)
-    if model is None:
-        raise ProjectorElementException(f"{collection} with id {id} does not exist")
-    return model
-
-
-async def get_models(
-    all_data_provider: ProjectorAllDataProvider, collection: str, ids: List[Any]
-) -> List[Dict[str, Any]]:
-    """
-    Tries to fetch all given models. Models are required to be all of the collection `collection`.
-    """
-    logger.info(
-        f"Note: a call to `get_models` with {collection}/{ids}. This might be cache-intensive"
-    )
-    return [await get_model(all_data_provider, collection, id) for id in ids]

server/openslides/utils/redis.py

@@ -10,15 +10,13 @@ logger = logging.getLogger(__name__)
 # Defaults
 use_redis = False
 use_read_only_redis = False
-read_only_redis_amount_replicas = None
-read_only_redis_wait_timeout = None
 
 try:
     import aioredis
 except ImportError:
     pass
 else:
-    from .redis_connection_pool import ConnectionPool
+    from .redis_connection_pool import ConnectionPool  # type: ignore
 
     # set use_redis to true, if there is a value for REDIS_ADDRESS in the settings
     redis_address = getattr(settings, "REDIS_ADDRESS", "")
@@ -33,13 +31,6 @@ else:
         if use_read_only_redis:
             logger.info(f"Redis read only address {redis_read_only_address}")
             read_only_pool = ConnectionPool({"address": redis_read_only_address})
-
-            read_only_redis_amount_replicas = getattr(settings, "AMOUNT_REPLICAS", 1)
-            logger.info(f"AMOUNT_REPLICAS={read_only_redis_amount_replicas}")
-            read_only_redis_wait_timeout = getattr(
-                settings, "REDIS_SLAVE_WAIT_TIMEOUT", 1000
-            )
-            logger.info(f"REDIS_SLAVE_WAIT_TIMEOUT={read_only_redis_wait_timeout}")
     else:
         logger.info("Redis is not configured.")
 

server/openslides/utils/redis_connection_pool.py

@@ -1,8 +1,10 @@
+# type: ignore
 import asyncio
+import sys
+import types
 from typing import Any, Dict, List, Optional
 
 import aioredis
-from channels_redis.core import ConnectionPool as ChannelRedisConnectionPool
 from django.conf import settings
 
 from . import logging
@@ -13,6 +15,128 @@ connection_pool_limit = getattr(settings, "CONNECTION_POOL_LIMIT", 100)
 logger.info(f"CONNECTION_POOL_LIMIT={connection_pool_limit}")
 
 
+# Copied from https://github.com/django/channels_redis/blob/master/channels_redis/core.py
+# and renamed..
+
+AIOREDIS_VERSION = tuple(map(int, aioredis.__version__.split(".")))
+
+
+def _wrap_close(loop, pool):
+    """
+    Decorate an event loop's close method with our own.
+    """
+    original_impl = loop.close
+
+    def _wrapper(self, *args, **kwargs):
+        # If the event loop was closed, there's nothing we can do anymore.
+        if not self.is_closed():
+            self.run_until_complete(pool.close_loop(self))
+        # Restore the original close() implementation after we're done.
+        self.close = original_impl
+        return self.close(*args, **kwargs)
+
+    loop.close = types.MethodType(_wrapper, loop)
+
+
+class ChannelRedisConnectionPool:
+    """
+    Connection pool manager for the channel layer.
+    It manages a set of connections for the given host specification and
+    taking into account asyncio event loops.
+    """
+
+    def __init__(self, host):
+        self.host = host
+        self.conn_map = {}
+        self.in_use = {}
+
+    def _ensure_loop(self, loop):
+        """
+        Get connection list for the specified loop.
+        """
+        if loop is None:
+            loop = asyncio.get_event_loop()
+
+        if loop not in self.conn_map:
+            # Swap the loop's close method with our own so we get
+            # a chance to do some cleanup.
+            _wrap_close(loop, self)
+            self.conn_map[loop] = []
+
+        return self.conn_map[loop], loop
+
+    async def pop(self, loop=None):
+        """
+        Get a connection for the given identifier and loop.
+        """
+        conns, loop = self._ensure_loop(loop)
+        if not conns:
+            if sys.version_info >= (3, 8, 0) and AIOREDIS_VERSION >= (1, 3, 1):
+                conn = await aioredis.create_redis(**self.host)
+            else:
+                conn = await aioredis.create_redis(**self.host, loop=loop)
+            conns.append(conn)
+        conn = conns.pop()
+        if conn.closed:
+            conn = await self.pop(loop=loop)
+            return conn
+        self.in_use[conn] = loop
+        return conn
+
+    def push(self, conn):
+        """
+        Return a connection to the pool.
+        """
+        loop = self.in_use[conn]
+        del self.in_use[conn]
+        if loop is not None:
+            conns, _ = self._ensure_loop(loop)
+            conns.append(conn)
+
+    def conn_error(self, conn):
+        """
+        Handle a connection that produced an error.
+        """
+        conn.close()
+        del self.in_use[conn]
+
+    def reset(self):
+        """
+        Clear all connections from the pool.
+        """
+        self.conn_map = {}
+        self.in_use = {}
+
+    async def close_loop(self, loop):
+        """
+        Close all connections owned by the pool on the given loop.
+        """
+        if loop in self.conn_map:
+            for conn in self.conn_map[loop]:
+                conn.close()
+                await conn.wait_closed()
+            del self.conn_map[loop]
+
+        for k, v in self.in_use.items():
+            if v is loop:
+                self.in_use[k] = None
+
+    async def close(self):
+        """
+        Close all connections owned by the pool.
+        """
+        conn_map = self.conn_map
+        in_use = self.in_use
+        self.reset()
+        for conns in conn_map.values():
+            for conn in conns:
+                conn.close()
+                await conn.wait_closed()
+        for conn in in_use:
+            conn.close()
+            await conn.wait_closed()
+
+
 class InvalidConnection(Exception):
     pass
 

server/openslides/utils/settings.py.tpl

@@ -83,45 +83,20 @@ DATABASES = {
 }
 
 
-# Set use_redis to True to activate redis as cache-, asgi- and session backend.
+# Collection Cache
-use_redis = False
+REDIS_ADDRESS = "redis://redis:6379/0"
 
-if use_redis:
+# Session backend
-    # Django Channels
+# Redis configuration for django-redis-sessions.
-
+# https://github.com/martinrusev/django-redis-sessions
-    # https://channels.readthedocs.io/en/latest/topics/channel_layers.html#configuration
+SESSION_ENGINE = 'redis_sessions.session'
-    CHANNEL_LAYERS = {
+SESSION_REDIS = {
-        "default": {
+    'host': 'redis',
-            "BACKEND": "channels_redis.core.RedisChannelLayer",
+    'port': 6379,
-            "CONFIG": {
+    'db': 0,
-                "hosts": [("localhost", 6379)],
+    'prefix': 'session',
-                "capacity": 100000,
+    'socket_timeout': 2
-            },
+}
-        },
-    }
-    # Collection Cache
-
-    # Can be:
-    # a Redis URI — "redis://host:6379/0?encoding=utf-8";
-    # a (host, port) tuple — ('localhost', 6379);
-    # or a unix domain socket path string — "/path/to/redis.sock".
-    REDIS_ADDRESS = "redis://127.0.0.1"
-    # REDIS_READ_ONLY_ADDRESS
-    AMOUNT_REPLICAS = 1
-
-    # Session backend
-
-    # Redis configuration for django-redis-sessions.
-    # https://github.com/martinrusev/django-redis-sessions
-
-    SESSION_ENGINE = 'redis_sessions.session'
-    SESSION_REDIS = {
-        'host': '127.0.0.1',
-        'port': 6379,
-        'db': 0,
-        'prefix': 'session',
-        'socket_timeout': 2
-    }
 
 # SAML integration
 # Please read https://github.com/OpenSlides/OpenSlides/blob/master/openslides/saml/README.md
@@ -144,21 +119,17 @@ ENABLE_ELECTRONIC_VOTING = False
 
 # Internationalization
 # https://docs.djangoproject.com/en/1.10/topics/i18n/
-
 TIME_ZONE = 'Europe/Berlin'
 
 
 # Static files (CSS, JavaScript, Images)
 # https://docs.djangoproject.com/en/1.10/howto/static-files/
-
 STATICFILES_DIRS = [os.path.join(OPENSLIDES_USER_DATA_DIR, 'static')] + STATICFILES_DIRS
-
 STATIC_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'collected-static')
 
 
 # Files
 # https://docs.djangoproject.com/en/1.10/topics/files/
-
 MEDIA_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'media', '')
 
 
@@ -169,7 +140,6 @@ MEDIA_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'media', '')
 
 # Logging
 # see https://docs.djangoproject.com/en/2.2/topics/logging/
-
 LOGGING = {
     'version': 1,
     'disable_existing_loggers': False,

server/openslides/utils/stream.py

@@ -0,0 +1,50 @@
+import json
+from typing import Any
+
+from django.conf import settings
+from typing_extensions import Protocol
+
+from . import logging
+from .redis import use_redis
+
+
+logger = logging.getLogger(__name__)
+
+REDIS_STREAM_MAXLEN = getattr(settings, "REDIS_STREAM_MAXLEN", None)
+logger.info(f"Redis stream maxlen {REDIS_STREAM_MAXLEN}")
+
+if use_redis:
+    from .redis import get_connection
+
+
+class Stream(Protocol):
+    def __init__(self) -> None:
+        ...
+
+    async def send(self, stream_name: str, payload: Any) -> None:
+        ...
+
+
+class RedisStream:
+    async def send(self, stream_name: str, payload: Any) -> None:
+        fields = {
+            "content": json.dumps(payload, separators=(",", ":")),
+        }
+        async with get_connection() as redis:
+            await redis.xadd(stream_name, fields, max_len=REDIS_STREAM_MAXLEN)
+
+
+class NoopStream:
+    async def send(self, stream_name: str, payload: Any) -> None:
+        pass
+
+
+def load_stream() -> Stream:
+    if use_redis:
+        return RedisStream()
+    else:
+        logger.error("You have to configure redis to let OpenSlides work properly!")
+        return NoopStream()
+
+
+stream = load_stream()