Merge pull request #5533 from FinnStutzenstein/externalAutoupdateService

External autoupdate service
This commit is contained in:
Finn Stutzenstein 2021-01-14 13:39:32 +01:00 committed by GitHub
commit b200cfbd07
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
112 changed files with 1849 additions and 4755 deletions

View File

@ -40,7 +40,7 @@ jobs:
run: mypy openslides/ tests/ run: mypy openslides/ tests/
- name: test using pytest - name: test using pytest
run: pytest --cov --cov-fail-under=75 run: pytest --cov --cov-fail-under=74
install-client-dependencies: install-client-dependencies:
runs-on: ubuntu-latest runs-on: ubuntu-latest

9
.gitignore vendored
View File

@ -18,14 +18,12 @@
.DS_Store .DS_Store
Thumbs.db Thumbs.db
# Virtual Environment # Virtual Environment
.virtualenv*/* .virtualenv*
.venv .venv
server/.venv
## Compatibility ## Compatibility
# OS4-Submodules # OS4-Submodules and aux-directories
/openslides-*/ /openslides-*/
/haproxy/
/docker/keys/ /docker/keys/
/docs/ /docs/
# OS3+-Submodules # OS3+-Submodules
@ -79,7 +77,8 @@ cypress.json
## Deployment ## Deployment
# Docker build artifacts # Docker build artifacts
/docker/docker-compose.yml docker/docker-compose.yml
*-version.txt *-version.txt
*.pem
# secrets # secrets
docker/secrets/*.env docker/secrets/*.env

3
.gitmodules vendored Normal file
View File

@ -0,0 +1,3 @@
[submodule "autoupdate"]
path = autoupdate
url = https://github.com/OpenSlides/openslides3-autoupdate-service.git

View File

@ -2,187 +2,82 @@
OpenSlides Development OpenSlides Development
======================== ========================
This instruction helps you to setup a development environment for OpenSlides. A Check requirements
simple dev setup will be configured without the need of the docker-compose
setup. There are only the server running without a cache and a sqlite database
and the client as an development server.
1. Installation on GNU/Linux or Mac OS X
----------------------------------------
a. Check requirements
''''''''''''''''''''' '''''''''''''''''''''
Make sure that you have installed `Python (>= 3.6) <https://www.python.org/>`_, - ``docker``
`Node.js (>=10.x) <https://nodejs.org/>`_ and `Git <http://git-scm.com/>`_ on - ``docker-compose``
your system. You also need build-essential packages and header files and a - ``git``
static library for Python. - ``make``
For Debian based systems (Ubuntu, etc) run:: Note about migrating from previous OpenSlides3 development
setups: You must set the ``OPENSLIDES_USER_DATA_DIR`` variable in
$ sudo apt-get install git nodejs npm build-essential python3-dev your ``server/personal_data/var/settings.py`` to ``'/app/personal_data/var'``
b. Get OpenSlides source code Get OpenSlides source code
''''''''''''''''''''''''''''' '''''''''''''''''''''''''''''
Clone current master version from `OpenSlides GitHub repository Clone current master version from `OpenSlides GitHub repository
<https://github.com/OpenSlides/OpenSlides/>`_:: <https://github.com/OpenSlides/OpenSlides/>`_::
$ git clone https://github.com/OpenSlides/OpenSlides.git git clone https://github.com/OpenSlides/OpenSlides.git
$ cd OpenSlides cd OpenSlides
TODO: submodules.
Start the development setup
''''''''''''''''''''''''''''''
make run-dev
c. Setup a virtual Python environment (optional) All you data (database, config, mediafiles) is stored in ``server/personal_data/var``.
''''''''''''''''''''''''''''''''''''''''''''''''
You can setup a virtual Python environment using the virtual environment Running the test cases
(venv) package for Python to install OpenSlides as non-root user. This will
allow for encapsulated dependencies. They will be installed in the virtual
environment and not globally on your system.
Setup and activate the virtual environment::
$ python3 -m venv .virtualenv
$ source .virtualenv/bin/activate
You can exit the environment with::
$ deactivate
d. Server
'''''''''
Go into the server's directory::
$ cd server/
Install all required Python packages::
$ pip install --upgrade setuptools pip
$ pip install --requirement requirements.txt
Create a settings file, run migrations and start the server::
$ python manage.py createsettings
$ python manage.py migrate
$ python manage.py runserver
All you data (database, config, mediafiles) are stored in ``personal_data/var``.
To get help on the command line options run::
$ python manage.py --help
Later you might want to restart the server with one of the following commands.
To run the OpenSlides server execute::
$ python manage.py runserver
When debugging something email related change the email backend to console::
$ python manage.py runserver --debug-email
The server is available under http://localhost:8000. Especially the rest interface
might be important during development: http://localhost:8000/rest/ (The trailing
slash is important!).
e. Client
'''''''''
Go in the client's directory::
$ cd client/
Install all dependencies and start the development server::
$ npm install
$ npm start
After a while, the client is available under http://localhost:4200.
2. Installation on Windows
--------------------------
Follow the instructions above (Installation on GNU/Linux or Mac OS X) but care
of the following variations.
To get Python download and run the latest `Python 3.7 32-bit (x86) executable
installer <https://www.python.org/downloads/windows/>`_. Note that the 32-bit
installer is required even on a 64-bit Windows system. If you use the 64-bit
installer, step d. of the instruction might fail unless you installed some
packages manually.
In some cases you have to install `MS Visual C++ 2015 build tools
<https://www.microsoft.com/en-us/download/details.aspx?id=48159>`_ before you
install the required python packages for OpenSlides (unfortunately Twisted
needs it).
To setup and activate the virtual environment in step c. use::
> .virtualenv\Scripts\activate.bat
All other commands are the same as for GNU/Linux and Mac OS X.
3. Running the test cases
------------------------- -------------------------
a. Running server tests For all services in submodules check out the documentation there.
Server tests andscripts
''''''''''''''''''''''' '''''''''''''''''''''''
You need to have python (>=3.8) and python-venv installed. Change your workdirectory to the server::
To run some server tests see `.travis.yml cd server
<https://github.com/OpenSlides/OpenSlides/blob/master/.travis.yml>`_.
b. Client tests and commands Setup an python virtual environment. If you have already done it, you can skip this step:
''''''''''''''''''''''''''''
Change to the client's directory to run every client related command. Run python3 -m venv .venv
client tests:: source .venv/bin/activate
pip install -U -r requirements.txt
$ npm test Make sure you are using the correct python version (e.g. try with explicit minor version: ``python3.8``). Activate it::
source .venv/bin/activate
To deactivate it type ``deactivate``. Running all tests and linters:
black openslides/ tests/
flake8 openslides/ tests/
mypy openslides/ tests/
isort -rc openslides/ tests/
pytest tests/
Client tests
''''''''''''
You need `node` and `npm` installed. Change to the client's directory. For the first time, install all dependencies::
cd client/
npm install
Run client tests::
npm test
Fix the code format and lint it with:: Fix the code format and lint it with::
$ npm run prettify-write npm run cleanup
$ npm run lint
To extract translations run:: To extract translations run::
$ npm run extract npm run extract
When updating, adding or changing used packages from npm, please update the
README.md using following command::
$ npm run licenses
4. Notes for running OpenSlides in larger setups
------------------------------------------------
For productive setups refer to the docker-compose setup described in the main
`README <https://github.com/OpenSlides/OpenSlides/blob/master/README.rst>`_.
While develpment it might be handy to use a cache and another database.
PostgreSQL is recommended and Redis necessary as a cache. Both can be set up in
the ``settings.py``. Please consider reading the `OpenSlides configuration
<https://github.com/OpenSlides/OpenSlides/blob/master/server/SETTINGS.rst>`_ page
to find out about all configurations, especially when using OpenSlides for big
assemblies.
If you followed the instructions and installed the pip requirements form the
``requirements.py`` all needed dependencies for another worker are installed.
Instead of running ``python manage.py runserver`` you can use daphne or gunicorn
(the latter is used in the prod setup)::
$ export DJANGO_SETTINGS_MODULE=settings
$ export PYTHONPATH=personal_data/var/
$ daphne -b 0.0.0.0 -p 8000 openslides.asgi:application
The last line may be interchangeable with gunicorn and uvicorn as protocol
server::
$ gunicorn -w 4 -b 0.0.0.0:8000 -k uvicorn.workers.UvicornWorker openslides.asgi:application

17
Makefile Normal file
View File

@ -0,0 +1,17 @@
build-dev:
make -C haproxy build-dev
git submodule foreach 'make build-dev'
docker-compose -f docker/docker-compose.dev.yml build
run-dev: | build-dev
UID=$$(id -u $${USER}) GID=$$(id -g $${USER}) docker-compose -f docker/docker-compose.dev.yml up
stop-dev:
docker-compose -f docker/docker-compose.dev.yml down
reload-haproxy:
docker-compose -f docker/docker-compose.dev.yml kill -s HUP haproxy
get-server-shell:
docker-compose -f docker/docker-compose.dev.yml run server bash

View File

@ -25,6 +25,8 @@ First, you have to clone this repository::
$ git clone https://github.com/OpenSlides/OpenSlides.git $ git clone https://github.com/OpenSlides/OpenSlides.git
$ cd OpenSlides/docker/ $ cd OpenSlides/docker/
TODO: submodules.
You need to build the Docker images for the client and server with this You need to build the Docker images for the client and server with this
script:: script::

1
autoupdate Submodule

@ -0,0 +1 @@
Subproject commit b187dd439bd7456e105901ca96cd0995862d4419

View File

@ -10,8 +10,8 @@ RUN npm install -g @angular/cli@^10
RUN ng config -g cli.warnings.versionMismatch false RUN ng config -g cli.warnings.versionMismatch false
USER openslides USER openslides
COPY package.json . COPY package.json package-lock.json ./
COPY package-lock.json . RUN npm -v
RUN npm ci RUN npm ci
COPY browserslist *.json ./ COPY browserslist *.json ./
COPY src ./src COPY src ./src

View File

@ -0,0 +1,11 @@
FROM node:13
WORKDIR /app
COPY package.json .
RUN npm install
RUN npm run postinstall
COPY . .
CMD npm start

View File

@ -25,29 +25,6 @@ http {
gzip_proxied expired no-cache no-store private auth; gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
location /apps {
proxy_pass http://server:8000;
}
location /media/ {
proxy_pass http://media:8000;
}
location /rest {
proxy_pass http://server:8000;
}
location /ws {
proxy_pass http://server:8000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
location /server-version.txt {
proxy_pass http://server:8000;
}
location = /basic_status {
stub_status;
}
location / { location / {
try_files $uri $uri/ /index.html; try_files $uri $uri/ /index.html;
} }

View File

@ -29,7 +29,7 @@
"dataGroups": [ "dataGroups": [
{ {
"name": "api", "name": "api",
"urls": ["/rest/*", "/apps/*"], "urls": ["/rest/*", "/apps/*", "/system/*", "/stats"],
"cacheConfig": { "cacheConfig": {
"maxSize": 0, "maxSize": 0,
"maxAge": "0u", "maxAge": "0u",

View File

@ -11,6 +11,7 @@
"scripts": { "scripts": {
"ng": "ng", "ng": "ng",
"start": "ng serve --proxy-config proxy.conf.json --host=0.0.0.0", "start": "ng serve --proxy-config proxy.conf.json --host=0.0.0.0",
"start-https": "ng serve --ssl --ssl-cert localhost.pem --ssl-key localhost-key.pem --proxy-config proxy.conf.json --host=0.0.0.0",
"start-es5": "ng serve --proxy-config proxy.conf.json --host=0.0.0.0 --configuration es5", "start-es5": "ng serve --proxy-config proxy.conf.json --host=0.0.0.0 --configuration es5",
"build": "ng build --prod", "build": "ng build --prod",
"build-to-dir": "npm run build -- --output-path", "build-to-dir": "npm run build -- --output-path",

View File

@ -15,5 +15,9 @@
"target": "ws://localhost:8000", "target": "ws://localhost:8000",
"secure": false, "secure": false,
"ws": true "ws": true
},
"/system/": {
"target": "https://localhost:8002",
"secure": false
} }
} }

View File

@ -10,10 +10,9 @@ import { CountUsersService } from './core/ui-services/count-users.service';
import { DataStoreUpgradeService } from './core/core-services/data-store-upgrade.service'; import { DataStoreUpgradeService } from './core/core-services/data-store-upgrade.service';
import { LoadFontService } from './core/ui-services/load-font.service'; import { LoadFontService } from './core/ui-services/load-font.service';
import { LoginDataService } from './core/ui-services/login-data.service'; import { LoginDataService } from './core/ui-services/login-data.service';
import { OfflineService } from './core/core-services/offline.service';
import { OperatorService } from './core/core-services/operator.service'; import { OperatorService } from './core/core-services/operator.service';
import { OverlayService } from './core/ui-services/overlay.service'; import { OverlayService } from './core/ui-services/overlay.service';
import { PingService } from './core/core-services/ping.service';
import { PrioritizeService } from './core/core-services/prioritize.service';
import { RoutingStateService } from './core/ui-services/routing-state.service'; import { RoutingStateService } from './core/ui-services/routing-state.service';
import { ServertimeService } from './core/core-services/servertime.service'; import { ServertimeService } from './core/core-services/servertime.service';
import { ThemeService } from './core/ui-services/theme.service'; import { ThemeService } from './core/ui-services/theme.service';
@ -75,17 +74,16 @@ export class AppComponent {
appRef: ApplicationRef, appRef: ApplicationRef,
servertimeService: ServertimeService, servertimeService: ServertimeService,
router: Router, router: Router,
offlineService: OfflineService,
operator: OperatorService, operator: OperatorService,
loginDataService: LoginDataService, loginDataService: LoginDataService,
constantsService: ConstantsService, // Needs to be started, so it can register itself to the WebsocketService constantsService: ConstantsService,
themeService: ThemeService, themeService: ThemeService,
overlayService: OverlayService, overlayService: OverlayService,
countUsersService: CountUsersService, // Needed to register itself. countUsersService: CountUsersService, // Needed to register itself.
configService: ConfigService, configService: ConfigService,
loadFontService: LoadFontService, loadFontService: LoadFontService,
dataStoreUpgradeService: DataStoreUpgradeService, // to start it. dataStoreUpgradeService: DataStoreUpgradeService, // to start it.
prioritizeService: PrioritizeService,
pingService: PingService,
routingState: RoutingStateService, routingState: RoutingStateService,
votingBannerService: VotingBannerService // needed for initialisation votingBannerService: VotingBannerService // needed for initialisation
) { ) {

View File

@ -0,0 +1,164 @@
import { EventEmitter, Injectable } from '@angular/core';
import { Observable, Subject } from 'rxjs';
import { AutoupdateFormat } from 'app/core/definitions/autoupdate-format';
import { trailingThrottleTime } from 'app/core/rxjs/trailing-throttle-time';
import { Identifiable } from 'app/shared/models/base/identifiable';
import { ConstantsService } from './constants.service';
import { HttpService } from './http.service';
interface ThrottleSettings {
AUTOUPDATE_DELAY?: number;
}
@Injectable({
providedIn: 'root'
})
export class AutoupdateThrottleService {
private readonly _autoupdatesToInject = new Subject<AutoupdateFormat>();
public get autoupdatesToInject(): Observable<AutoupdateFormat> {
return this._autoupdatesToInject.asObservable();
}
private readonly receivedAutoupdate = new EventEmitter<void>();
private pendingAutoupdates = [];
private disabledUntil: number | null = null;
private delay = 0;
private maxSeenChangeId = 0;
private get isActive(): boolean {
return this.delay !== 0;
}
public constructor(private constantsService: ConstantsService, private httpService: HttpService) {
this.constantsService.get<ThrottleSettings>('Settings').subscribe(settings => {
// This is a one-shot. If the delay was set one time >0, it cannot be changed afterwards.
// A change is more complicated since you have to unsubscribe, clean pending autoupdates,
// subscribe again and make sure, that no autoupdate is missed.
if (this.delay === 0 && settings.AUTOUPDATE_DELAY) {
this.delay = 1000 * settings.AUTOUPDATE_DELAY;
console.log(`Configured autoupdate delay: ${this.delay}ms`);
this.receivedAutoupdate
.pipe(trailingThrottleTime(this.delay))
.subscribe(() => this.processPendingAutoupdates());
} else if (this.delay === 0) {
console.log('No autoupdate delay');
}
});
this.httpService.responseChangeIds.subscribe(changeId => this.disableUntil(changeId));
}
public newAutoupdate(autoupdate: AutoupdateFormat): void {
if (autoupdate.to_change_id > this.maxSeenChangeId) {
this.maxSeenChangeId = autoupdate.to_change_id;
}
if (!this.isActive) {
this._autoupdatesToInject.next(autoupdate);
} else if (this.disabledUntil !== null) {
this._autoupdatesToInject.next(autoupdate);
if (autoupdate.to_change_id >= this.disabledUntil) {
this.disabledUntil = null;
console.log('Throttling autoupdates again');
}
} else {
this.pendingAutoupdates.push(autoupdate);
this.receivedAutoupdate.emit();
}
}
public disableUntil(changeId: number): void {
// Wait for an autoupdate with to_id >= changeId.
console.log(this.delay);
if (!this.isActive) {
return;
}
this.processPendingAutoupdates();
// Checking with maxSeenChangeId is for the following race condition:
// If the autoupdate comes before the response, it must not be throttled.
// But flushing pending autoupdates is important since *if* the autoupdate
// was early, it is in the pending queue.
if (changeId <= this.maxSeenChangeId) {
return;
}
console.log('Disable autoupdate until change id', changeId);
this.disabledUntil = changeId;
}
private processPendingAutoupdates(): void {
const autoupdates = this.pendingAutoupdates;
if (autoupdates.length === 0) {
return;
}
this.pendingAutoupdates = [];
const autoupdate = this.mergeAutoupdates(autoupdates);
this._autoupdatesToInject.next(autoupdate);
}
private mergeAutoupdates(autoupdates: AutoupdateFormat[]): AutoupdateFormat {
const mergedAutoupdate: AutoupdateFormat = {
changed: {},
deleted: {},
from_change_id: autoupdates[0].from_change_id,
to_change_id: autoupdates[autoupdates.length - 1].to_change_id,
all_data: false
};
let lastToChangeId = null;
for (const au of autoupdates) {
if (lastToChangeId === null) {
lastToChangeId = au.to_change_id;
} else {
if (au.from_change_id !== lastToChangeId) {
console.warn('!!!', autoupdates, au);
}
lastToChangeId = au.to_change_id;
}
this.applyAutoupdate(au, mergedAutoupdate);
}
return mergedAutoupdate;
}
private applyAutoupdate(from: AutoupdateFormat, into: AutoupdateFormat): void {
if (from.all_data) {
into.all_data = true;
into.changed = from.changed;
into.deleted = from.deleted;
return;
}
for (const collection of Object.keys(from.deleted)) {
for (const id of from.deleted[collection]) {
if (into.changed[collection]) {
into.changed[collection] = into.changed[collection].filter(obj => (obj as Identifiable).id !== id);
}
if (!into.deleted[collection]) {
into.deleted[collection] = [];
}
into.deleted[collection].push(id);
}
}
for (const collection of Object.keys(from.changed)) {
for (const obj of from.changed[collection]) {
if (into.deleted[collection]) {
into.deleted[collection] = into.deleted[collection].filter(id => id !== (obj as Identifiable).id);
}
if (!into.changed[collection]) {
into.changed[collection] = [];
}
into.changed[collection].push(obj);
}
}
}
}

View File

@ -1,57 +1,16 @@
import { Injectable } from '@angular/core'; import { Injectable } from '@angular/core';
import { AutoupdateFormat } from '../definitions/autoupdate-format';
import { AutoupdateThrottleService } from './autoupdate-throttle.service';
import { BaseModel } from '../../shared/models/base/base-model'; import { BaseModel } from '../../shared/models/base/base-model';
import { CollectionStringMapperService } from './collection-string-mapper.service'; import { CollectionStringMapperService } from './collection-string-mapper.service';
import { CommunicationManagerService, OfflineError } from './communication-manager.service';
import { DataStoreService, DataStoreUpdateManagerService } from './data-store.service'; import { DataStoreService, DataStoreUpdateManagerService } from './data-store.service';
import { HttpService } from './http.service';
import { Mutex } from '../promises/mutex'; import { Mutex } from '../promises/mutex';
import { WebsocketService, WEBSOCKET_ERROR_CODES } from './websocket.service';
export interface AutoupdateFormat {
/**
* All changed (and created) items as their full/restricted data grouped by their collection.
*/
changed: {
[collectionString: string]: object[];
};
/** /**
* All deleted items (by id) grouped by their collection. * Handles the initial update and automatic updates
*/
deleted: {
[collectionString: string]: number[];
};
/**
* The lower change id bond for this autoupdate
*/
from_change_id: number;
/**
* The upper change id bound for this autoupdate
*/
to_change_id: number;
/**
* Flag, if this autoupdate contains all data. If so, the DS needs to be resetted.
*/
all_data: boolean;
}
export function isAutoupdateFormat(obj: any): obj is AutoupdateFormat {
const format = obj as AutoupdateFormat;
return (
obj &&
typeof obj === 'object' &&
format.changed !== undefined &&
format.deleted !== undefined &&
format.from_change_id !== undefined &&
format.to_change_id !== undefined &&
format.all_data !== undefined
);
}
/**
* Handles the initial update and automatic updates using the {@link WebsocketService}
* Incoming objects, usually BaseModels, will be saved in the dataStore (`this.DS`) * Incoming objects, usually BaseModels, will be saved in the dataStore (`this.DS`)
* This service usually creates all models * This service usually creates all models
*/ */
@ -61,32 +20,49 @@ export function isAutoupdateFormat(obj: any): obj is AutoupdateFormat {
export class AutoupdateService { export class AutoupdateService {
private mutex = new Mutex(); private mutex = new Mutex();
/** private streamCloseFn: () => void | null = null;
* Constructor to create the AutoupdateService. Calls the constructor of the parent class.
* @param websocketService private lastMessageContainedAllData = false;
* @param DS
* @param modelMapper
*/
public constructor( public constructor(
private websocketService: WebsocketService,
private DS: DataStoreService, private DS: DataStoreService,
private modelMapper: CollectionStringMapperService, private modelMapper: CollectionStringMapperService,
private DSUpdateManager: DataStoreUpdateManagerService private DSUpdateManager: DataStoreUpdateManagerService,
private communicationManager: CommunicationManagerService,
private autoupdateThrottle: AutoupdateThrottleService
) { ) {
this.websocketService.getOberservable<AutoupdateFormat>('autoupdate').subscribe(response => { this.communicationManager.startCommunicationEvent.subscribe(() => this.startAutoupdate());
this.storeResponse(response);
});
// Check for too high change id-errors. If this happens, reset the DS and get fresh data. this.autoupdateThrottle.autoupdatesToInject.subscribe(autoupdate => this.storeAutoupdate(autoupdate));
this.websocketService.errorResponseObservable.subscribe(error => { }
if (error.code === WEBSOCKET_ERROR_CODES.CHANGE_ID_TOO_HIGH) {
this.doFullUpdate(); public async startAutoupdate(changeId?: number): Promise<void> {
this.stopAutoupdate();
try {
this.streamCloseFn = await this.communicationManager.subscribe<AutoupdateFormat>(
'/system/autoupdate',
autoupdate => {
this.autoupdateThrottle.newAutoupdate(autoupdate);
},
() => ({ change_id: (changeId ? changeId : this.DS.maxChangeId).toString() })
);
} catch (e) {
if (!(e instanceof OfflineError)) {
console.error(e);
}
}
}
public stopAutoupdate(): void {
if (this.streamCloseFn) {
this.streamCloseFn();
this.streamCloseFn = null;
} }
});
} }
/** /**
* Handle the answer of incoming data via {@link WebsocketService}. * Handle the answer of incoming data, after it was throttled.
* *
* Detects the Class of an incomming model, creates a new empty object and assigns * Detects the Class of an incomming model, creates a new empty object and assigns
* the data to it using the deserialize function. Also models that are flagged as deleted * the data to it using the deserialize function. Also models that are flagged as deleted
@ -94,8 +70,9 @@ export class AutoupdateService {
* *
* Handles the change ids of all autoupdates. * Handles the change ids of all autoupdates.
*/ */
private async storeResponse(autoupdate: AutoupdateFormat): Promise<void> { private async storeAutoupdate(autoupdate: AutoupdateFormat): Promise<void> {
const unlock = await this.mutex.lock(); const unlock = await this.mutex.lock();
this.lastMessageContainedAllData = autoupdate.all_data;
if (autoupdate.all_data) { if (autoupdate.all_data) {
await this.storeAllData(autoupdate); await this.storeAllData(autoupdate);
} else { } else {
@ -138,17 +115,10 @@ export class AutoupdateService {
} else { } else {
// autoupdate fully in the future. we are missing something! // autoupdate fully in the future. we are missing something!
console.log('Autoupdate in the future', maxChangeId, autoupdate.from_change_id, autoupdate.to_change_id); console.log('Autoupdate in the future', maxChangeId, autoupdate.from_change_id, autoupdate.to_change_id);
this.requestChanges(); this.startAutoupdate(); // restarts it.
} }
} }
public async injectAutoupdateIgnoreChangeId(autoupdate: AutoupdateFormat): Promise<void> {
const unlock = await this.mutex.lock();
console.debug('inject autoupdate', autoupdate);
await this.injectAutupdateIntoDS(autoupdate, false);
unlock();
}
private async injectAutupdateIntoDS(autoupdate: AutoupdateFormat, flush: boolean): Promise<void> { private async injectAutupdateIntoDS(autoupdate: AutoupdateFormat, flush: boolean): Promise<void> {
const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this.DS); const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this.DS);
@ -187,35 +157,21 @@ export class AutoupdateService {
} }
} }
/**
* Sends a WebSocket request to the Server with the maxChangeId of the DataStore.
* The server should return an autoupdate with all new data.
*/
public requestChanges(): void {
console.log(`requesting changed objects with DS max change id ${this.DS.maxChangeId}`);
this.websocketService.send('getElements', { change_id: this.DS.maxChangeId });
}
/** /**
* Does a full update: Requests all data from the server and sets the DS to the fresh data. * Does a full update: Requests all data from the server and sets the DS to the fresh data.
*/ */
public async doFullUpdate(): Promise<void> { public async doFullUpdate(): Promise<void> {
const oldChangeId = this.DS.maxChangeId; if (this.lastMessageContainedAllData) {
const response = await this.websocketService.sendAndGetResponse<{}, AutoupdateFormat>('getElements', {}); console.log('full update requested. Skipping, last message already contained all data');
const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this.DS);
let allModels: BaseModel[] = [];
for (const collection of Object.keys(response.changed)) {
if (this.modelMapper.isCollectionRegistered(collection)) {
allModels = allModels.concat(this.mapObjectsToBaseModels(collection, response.changed[collection]));
} else { } else {
console.error(`Unregistered collection "${collection}". Ignore it.`); console.log('requesting full update.');
// The mutex is needed, so the DS is not cleared, if there is
// another autoupdate running.
const unlock = await this.mutex.lock();
this.stopAutoupdate();
await this.DS.clear();
this.startAutoupdate();
unlock();
} }
} }
await this.DS.set(allModels, response.to_change_id);
this.DSUpdateManager.commit(updateSlot, response.to_change_id, true);
console.log(`Full update done from ${oldChangeId} to ${response.to_change_id}`);
}
} }

View File

@ -0,0 +1,186 @@
import { HttpParams } from '@angular/common/http';
import { EventEmitter, Injectable } from '@angular/core';
import { Observable } from 'rxjs';
import { HttpService } from './http.service';
import { OfflineBroadcastService, OfflineReason } from './offline-broadcast.service';
import { OperatorService } from './operator.service';
import { SleepPromise } from '../promises/sleep';
import {
CommunicationError,
ErrorType,
Stream,
StreamContainer,
StreamingCommunicationService,
verboseErrorType
} from './streaming-communication.service';
type HttpParamsGetter = () => HttpParams | { [param: string]: string | string[] };
const MAX_STREAM_FAILURE_RETRIES = 3;
export class OfflineError extends Error {
public constructor() {
super('');
this.name = 'OfflineError';
}
}
interface StreamConnectionWrapper {
id: number;
url: string;
messageHandler: (message: any) => void;
params: HttpParamsGetter;
stream?: Stream<any>;
hasErroredAmount: number;
}
@Injectable({
providedIn: 'root'
})
export class CommunicationManagerService {
private communicationAllowed = false;
private readonly _startCommunicationEvent = new EventEmitter<void>();
public get startCommunicationEvent(): Observable<void> {
return this._startCommunicationEvent.asObservable();
}
private readonly _stopCommunicationEvent = new EventEmitter<void>();
public get stopCommunicationEvent(): Observable<void> {
return this._stopCommunicationEvent.asObservable();
}
private streamContainers: { [id: number]: StreamContainer } = {};
public constructor(
private streamingCommunicationService: StreamingCommunicationService,
private offlineBroadcastService: OfflineBroadcastService,
private http: HttpService,
private operatorService: OperatorService
) {
this.offlineBroadcastService.goOfflineObservable.subscribe(() => this.closeConnections());
}
public async subscribe<T>(
url: string,
messageHandler: (message: T) => void,
params?: HttpParamsGetter
): Promise<() => void> {
if (!params) {
params = () => null;
}
const streamContainer = new StreamContainer(url, messageHandler, params);
return await this.connectWithWrapper(streamContainer);
}
public startCommunication(): void {
if (this.communicationAllowed) {
console.error('Illegal state! Do not emit this event multiple times');
} else {
this.communicationAllowed = true;
this._startCommunicationEvent.emit();
}
}
private async connectWithWrapper(streamContainer: StreamContainer): Promise<() => void> {
console.log('connect', streamContainer, streamContainer.stream);
const errorHandler = (type: ErrorType, error: CommunicationError, message: string) =>
this.handleError(streamContainer, type, error, message);
this.streamingCommunicationService.subscribe(streamContainer, errorHandler);
this.streamContainers[streamContainer.id] = streamContainer;
return () => this.close(streamContainer);
}
private async handleError(
streamContainer: StreamContainer,
type: ErrorType,
error: CommunicationError,
message: string
): Promise<void> {
console.log('handle Error', streamContainer, streamContainer.stream, verboseErrorType(type), error, message);
streamContainer.stream.close();
streamContainer.stream = null;
streamContainer.hasErroredAmount++;
if (streamContainer.hasErroredAmount > MAX_STREAM_FAILURE_RETRIES) {
this.goOffline(streamContainer, OfflineReason.ConnectionLost);
} else if (type === ErrorType.Client && error.type === 'ErrorAuth') {
this.goOffline(streamContainer, OfflineReason.WhoAmIFailed);
} else {
// retry it after some time:
console.log(
`Retry no. ${streamContainer.hasErroredAmount} of ${MAX_STREAM_FAILURE_RETRIES} for ${streamContainer.url}`
);
try {
await this.delayAndCheckReconnection();
await this.connectWithWrapper(streamContainer);
} catch (e) {
// delayAndCheckReconnection can throw an OfflineError,
// which are just an 'abord mission' signal. Here, those errors can be ignored.
}
}
}
private async delayAndCheckReconnection(): Promise<void> {
const delay = Math.floor(Math.random() * 3000 + 2000);
console.log(`retry again in ${delay} ms`);
await SleepPromise(delay);
// do not continue, if we are offline!
if (this.offlineBroadcastService.isOffline()) {
console.log('we are offline?');
throw new OfflineError();
}
// do not continue, if we are offline!
if (!this.shouldRetryConnecting()) {
console.log('operator changed, do not rety');
throw new OfflineError(); // TODO: This error is not really good....
}
}
public closeConnections(): void {
for (const streamWrapper of Object.values(this.streamContainers)) {
if (streamWrapper.stream) {
streamWrapper.stream.close();
}
}
this.streamContainers = {};
this.communicationAllowed = false;
this._stopCommunicationEvent.emit();
}
private goOffline(streamContainer: StreamContainer, reason: OfflineReason): void {
delete this.streamContainers[streamContainer.id];
this.closeConnections(); // here we close the connections early.
this.offlineBroadcastService.goOffline(reason);
}
private close(streamConnectionWrapper: StreamConnectionWrapper): void {
if (this.streamContainers[streamConnectionWrapper.id]) {
this.streamContainers[streamConnectionWrapper.id].stream.close();
delete this.streamContainers[streamConnectionWrapper.id];
}
}
// Checks the operator: If we do not have a valid user,
// do not even try to connect again..
private shouldRetryConnecting(): boolean {
return this.operatorService.guestsEnabled || !!this.operatorService.user;
}
public async isCommunicationServiceOnline(): Promise<boolean> {
try {
const response = await this.http.get<{ healthy: boolean }>('/system/health');
return !!response.healthy;
} catch (e) {
return false;
}
}
}

View File

@ -1,9 +1,11 @@
import { Injectable } from '@angular/core'; import { Injectable } from '@angular/core';
import { environment } from 'environments/environment';
import { BehaviorSubject, Observable } from 'rxjs'; import { BehaviorSubject, Observable } from 'rxjs';
import { filter } from 'rxjs/operators'; import { filter } from 'rxjs/operators';
import { WebsocketService } from './websocket.service'; import { CommunicationManagerService } from './communication-manager.service';
import { HttpService } from './http.service';
/** /**
* constants have a key associated with the data. * constants have a key associated with the data.
@ -36,24 +38,15 @@ export class ConstantsService {
*/ */
private subjects: { [key: string]: BehaviorSubject<any> } = {}; private subjects: { [key: string]: BehaviorSubject<any> } = {};
/** public constructor(communicationManager: CommunicationManagerService, private http: HttpService) {
* @param websocketService communicationManager.startCommunicationEvent.subscribe(async () => {
*/ console.log('start communication');
public constructor(private websocketService: WebsocketService) { this.constants = await this.http.get<Constants>(environment.urlPrefix + '/core/constants/');
// The hook for recieving constants. console.log('constants:', this.constants);
websocketService.getOberservable<Constants>('constants').subscribe(constants => {
this.constants = constants;
Object.keys(this.subjects).forEach(key => { Object.keys(this.subjects).forEach(key => {
this.subjects[key].next(this.constants[key]); this.subjects[key].next(this.constants[key]);
}); });
}); });
// We can request constants, if the websocket connection opens.
// On retries, the `refresh()` method is called by the OpenSlidesService, so
// here we do not need to take care about this.
websocketService.noRetryConnectEvent.subscribe(() => {
this.refresh();
});
} }
/** /**
@ -66,14 +59,4 @@ export class ConstantsService {
} }
return this.subjects[key].asObservable().pipe(filter(x => !!x)); return this.subjects[key].asObservable().pipe(filter(x => !!x));
} }
/**
* Refreshed the constants
*/
public refresh(): Promise<void> {
if (!this.websocketService.isConnected) {
return;
}
this.websocketService.send('constants', {});
}
} }

View File

@ -361,12 +361,12 @@ export class DataStoreService {
* *
* @returns The max change id. * @returns The max change id.
*/ */
public async initFromStorage(): Promise<number> { public async initFromStorage(): Promise<void> {
// This promise will be resolved with cached datastore. // This promise will be resolved with cached datastore.
const store = await this.storageService.get<JsonStorage>(DataStoreService.cachePrefix + 'DS'); const store = await this.storageService.get<JsonStorage>(DataStoreService.cachePrefix + 'DS');
if (!store) { if (!store) {
await this.clear(); await this.clear();
return this.maxChangeId; return;
} }
const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this); const updateSlot = await this.DSUpdateManager.getNewUpdateSlot(this);
@ -395,7 +395,6 @@ export class DataStoreService {
this.DSUpdateManager.dropUpdateSlot(); this.DSUpdateManager.dropUpdateSlot();
await this.clear(); await this.clear();
} }
return this.maxChangeId;
} }
/** /**
@ -670,6 +669,6 @@ export class DataStoreService {
public print(): void { public print(): void {
console.log('Max change id', this.maxChangeId); console.log('Max change id', this.maxChangeId);
console.log(JSON.stringify(this.jsonStore)); console.log(JSON.stringify(this.jsonStore));
console.log(this.modelStore); console.log(JSON.parse(JSON.stringify(this.modelStore)));
} }
} }

View File

@ -2,22 +2,14 @@ import { HttpClient, HttpErrorResponse, HttpHeaders } from '@angular/common/http
import { Injectable } from '@angular/core'; import { Injectable } from '@angular/core';
import { TranslateService } from '@ngx-translate/core'; import { TranslateService } from '@ngx-translate/core';
import { Subject } from 'rxjs';
import { AutoupdateFormat, AutoupdateService, isAutoupdateFormat } from './autoupdate.service'; import { AutoupdateFormat } from '../definitions/autoupdate-format';
import { AutoupdateThrottleService } from './autoupdate-throttle.service';
import { HTTPMethod } from '../definitions/http-methods';
import { OpenSlidesStatusService } from './openslides-status.service'; import { OpenSlidesStatusService } from './openslides-status.service';
import { formatQueryParams, QueryParams } from '../definitions/query-params'; import { formatQueryParams, QueryParams } from '../definitions/query-params';
/**
* Enum for different HTTPMethods
*/
export enum HTTPMethod {
GET = 'get',
POST = 'post',
PUT = 'put',
PATCH = 'patch',
DELETE = 'delete'
}
export interface ErrorDetailResponse { export interface ErrorDetailResponse {
detail: string | string[]; detail: string | string[];
args?: string[]; args?: string[];
@ -33,12 +25,12 @@ function isErrorDetailResponse(obj: any): obj is ErrorDetailResponse {
} }
interface AutoupdateResponse { interface AutoupdateResponse {
autoupdate: AutoupdateFormat; change_id: number;
data?: any; data?: any;
} }
function isAutoupdateReponse(obj: any): obj is AutoupdateResponse { function isAutoupdateReponse(obj: any): obj is AutoupdateResponse {
return obj && typeof obj === 'object' && isAutoupdateFormat((obj as AutoupdateResponse).autoupdate); return obj && typeof obj === 'object' && typeof (obj as AutoupdateResponse).change_id === 'number';
} }
/** /**
@ -53,6 +45,8 @@ export class HttpService {
*/ */
private defaultHeaders: HttpHeaders; private defaultHeaders: HttpHeaders;
public readonly responseChangeIds = new Subject<number>();
/** /**
* Construct a HttpService * Construct a HttpService
* *
@ -65,8 +59,7 @@ export class HttpService {
public constructor( public constructor(
private http: HttpClient, private http: HttpClient,
private translate: TranslateService, private translate: TranslateService,
private OSStatus: OpenSlidesStatusService, private OSStatus: OpenSlidesStatusService
private autoupdateService: AutoupdateService
) { ) {
this.defaultHeaders = new HttpHeaders().set('Content-Type', 'application/json'); this.defaultHeaders = new HttpHeaders().set('Content-Type', 'application/json');
} }
@ -205,8 +198,8 @@ export class HttpService {
private processResponse<T>(responseData: T): T { private processResponse<T>(responseData: T): T {
if (isAutoupdateReponse(responseData)) { if (isAutoupdateReponse(responseData)) {
this.autoupdateService.injectAutoupdateIgnoreChangeId(responseData.autoupdate); this.responseChangeIds.next(responseData.change_id);
responseData = responseData.data; return responseData.data;
} }
return responseData; return responseData;
} }

View File

@ -2,8 +2,9 @@ import { Injectable } from '@angular/core';
import { Observable, Subject } from 'rxjs'; import { Observable, Subject } from 'rxjs';
import { CommunicationManagerService, OfflineError } from './communication-manager.service';
import { HttpService } from './http.service';
import { OperatorService } from './operator.service'; import { OperatorService } from './operator.service';
import { WebsocketService } from './websocket.service';
/** /**
* Encapslates the name and content of every message regardless of being a request or response. * Encapslates the name and content of every message regardless of being a request or response.
@ -17,7 +18,12 @@ interface NotifyBase<T> {
/** /**
* The content to send. * The content to send.
*/ */
content: T; message: T;
}
function isNotifyBase(obj: object): obj is NotifyResponse<any> {
const base = obj as NotifyBase<any>;
return !!obj && base.message !== undefined && base.name !== undefined;
} }
/** /**
@ -26,15 +32,18 @@ interface NotifyBase<T> {
* channel names. * channel names.
*/ */
export interface NotifyRequest<T> extends NotifyBase<T> { export interface NotifyRequest<T> extends NotifyBase<T> {
channel_id: string;
to_all?: boolean;
/** /**
* User ids (or `true` for all users) to send this message to. * User ids (or `true` for all users) to send this message to.
*/ */
users?: number[] | boolean; to_users?: number[];
/** /**
* An array of channels to send this message to. * An array of channels to send this message to.
*/ */
replyChannels?: string[]; to_channels?: string[];
} }
/** /**
@ -45,12 +54,12 @@ export interface NotifyResponse<T> extends NotifyBase<T> {
* This is the channel name of the one, who sends this message. Can be use to directly * This is the channel name of the one, who sends this message. Can be use to directly
* answer this message. * answer this message.
*/ */
senderChannelName: string; sender_channel_id: string;
/** /**
* The user id of the user who sends this message. It is 0 for Anonymous. * The user id of the user who sends this message. It is 0 for Anonymous.
*/ */
senderUserId: number; sender_user_id: number;
/** /**
* This is validated here and is true, if the senderUserId matches the current operator's id. * This is validated here and is true, if the senderUserId matches the current operator's id.
@ -59,6 +68,20 @@ export interface NotifyResponse<T> extends NotifyBase<T> {
sendByThisUser: boolean; sendByThisUser: boolean;
} }
function isNotifyResponse(obj: object): obj is NotifyResponse<any> {
const response = obj as NotifyResponse<any>;
// Note: we do not test for sendByThisUser, since it is set later in our code.
return isNotifyBase(obj) && response.sender_channel_id !== undefined && response.sender_user_id !== undefined;
}
interface ChannelIdResponse {
channel_id: string;
}
function isChannelIdResponse(obj: object): obj is ChannelIdResponse {
return !!obj && (obj as ChannelIdResponse).channel_id !== undefined;
}
/** /**
* Handles all incoming and outgoing notify messages via {@link WebsocketService}. * Handles all incoming and outgoing notify messages via {@link WebsocketService}.
*/ */
@ -78,18 +101,41 @@ export class NotifyService {
[name: string]: Subject<NotifyResponse<any>>; [name: string]: Subject<NotifyResponse<any>>;
} = {}; } = {};
/** private channelId: string;
* Constructor to create the NotifyService. Registers itself to the WebsocketService.
* @param websocketService public constructor(
*/ private communicationManager: CommunicationManagerService,
public constructor(private websocketService: WebsocketService, private operator: OperatorService) { private http: HttpService,
websocketService.getOberservable<NotifyResponse<any>>('notify').subscribe(notify => { private operator: OperatorService
notify.sendByThisUser = notify.senderUserId === (this.operator.user ? this.operator.user.id : 0); ) {
this.communicationManager.startCommunicationEvent.subscribe(() => this.startListening());
this.communicationManager.stopCommunicationEvent.subscribe(() => (this.channelId = null));
}
private async startListening(): Promise<void> {
try {
await this.communicationManager.subscribe<NotifyResponse<any> | ChannelIdResponse>(
'/system/notify',
notify => {
if (isChannelIdResponse(notify)) {
this.channelId = notify.channel_id;
} else if (isNotifyResponse(notify)) {
notify.sendByThisUser =
notify.sender_user_id === (this.operator.user ? this.operator.user.id : 0);
this.notifySubject.next(notify); this.notifySubject.next(notify);
if (this.messageSubjects[notify.name]) { if (this.messageSubjects[notify.name]) {
this.messageSubjects[notify.name].next(notify); this.messageSubjects[notify.name].next(notify);
} }
}); } else {
console.error('Unknwon notify message', notify);
}
}
);
} catch (e) {
if (!(e instanceof OfflineError)) {
console.log(e);
}
}
} }
/** /**
@ -97,8 +143,8 @@ export class NotifyService {
* @param name The name of the notify message * @param name The name of the notify message
* @param content The payload to send * @param content The payload to send
*/ */
public sendToAllUsers<T>(name: string, content: T): void { public async sendToAllUsers<T>(name: string, content: T): Promise<void> {
this.send(name, content); await this.send(name, content, true);
} }
/** /**
@ -107,8 +153,11 @@ export class NotifyService {
* @param content The payload to send. * @param content The payload to send.
* @param users Multiple user ids. * @param users Multiple user ids.
*/ */
public sendToUsers<T>(name: string, content: T, ...users: number[]): void { public async sendToUsers<T>(name: string, content: T, ...users: number[]): Promise<void> {
this.send(name, content, users); if (users.length < 1) {
throw new Error('You have to provide at least one user');
}
await this.send(name, content, false, users);
} }
/** /**
@ -117,35 +166,48 @@ export class NotifyService {
* @param content The payload to send. * @param content The payload to send.
* @param channels Multiple channels to send this message to. * @param channels Multiple channels to send this message to.
*/ */
public sendToChannels<T>(name: string, content: T, ...channels: string[]): void { public async sendToChannels<T>(name: string, content: T, ...channels: string[]): Promise<void> {
if (channels.length < 1) { if (channels.length < 1) {
throw new Error('You have to provide at least one channel'); throw new Error('You have to provide at least one channel');
} }
this.send(name, content, null, channels); await this.send(name, content, false, null, channels);
} }
/** /**
* General send function for notify messages. * General send function for notify messages.
* @param name The name of the notify message * @param name The name of the notify message
* @param content The payload to send. * @param message The payload to send.
* @param users Either an array of IDs or `true` meaning of sending this message to all online users clients. * @param users Either an array of IDs or `true` meaning of sending this message to all online users clients.
* @param channels An array of channels to send this message to. * @param channels An array of channels to send this message to.
*/ */
public send<T>(name: string, content: T, users?: number[] | boolean, channels?: string[]): void { private async send<T>(
name: string,
message: T,
toAll?: boolean,
users?: number[],
channels?: string[]
): Promise<void> {
if (!this.channelId) {
throw new Error('No channel id!');
}
const notify: NotifyRequest<T> = { const notify: NotifyRequest<T> = {
name: name, name: name,
content: content message: message,
channel_id: this.channelId
}; };
if (typeof users === 'boolean' && users !== true) { if (toAll === true) {
throw new Error('You just can give true as a boolean to send this message to all users.'); notify.to_all = true;
} }
if (users !== null) { if (users) {
notify.users = users; notify.to_users = users;
} }
if (channels !== null) { if (channels) {
notify.replyChannels = channels; notify.to_channels = channels;
} }
this.websocketService.send('notify', notify);
console.debug('send notify', notify);
await this.http.post<unknown>('/system/notify/send', notify);
} }
/** /**

View File

@ -0,0 +1,37 @@
import { EventEmitter, Injectable } from '@angular/core';
import { BehaviorSubject, Observable } from 'rxjs';
export enum OfflineReason {
WhoAmIFailed,
ConnectionLost
}
@Injectable({
providedIn: 'root'
})
export class OfflineBroadcastService {
public readonly isOfflineSubject = new BehaviorSubject<boolean>(false);
public get isOfflineObservable(): Observable<boolean> {
return this.isOfflineSubject.asObservable();
}
private readonly _goOffline = new EventEmitter<OfflineReason>();
public get goOfflineObservable(): Observable<OfflineReason> {
return this._goOffline.asObservable();
}
public constructor() {}
public goOffline(reason: OfflineReason): void {
this._goOffline.emit(reason);
}
public isOffline(): boolean {
return this.isOfflineSubject.getValue();
}
public isOnline(): boolean {
return !this.isOffline();
}
}

View File

@ -1,10 +1,9 @@
import { Injectable } from '@angular/core'; import { Injectable } from '@angular/core';
import { marker as _ } from '@biesbjerg/ngx-translate-extract-marker'; import { CommunicationManagerService } from './communication-manager.service';
import { TranslateService } from '@ngx-translate/core'; import { OfflineBroadcastService, OfflineReason } from './offline-broadcast.service';
import { BehaviorSubject, Observable } from 'rxjs'; import { OpenSlidesService } from './openslides.service';
import { OperatorService, WhoAmI } from './operator.service';
import { BannerDefinition, BannerService } from '../ui-services/banner.service';
/** /**
* This service handles everything connected with being offline. * This service handles everything connected with being offline.
@ -16,63 +15,111 @@ import { BannerDefinition, BannerService } from '../ui-services/banner.service';
providedIn: 'root' providedIn: 'root'
}) })
export class OfflineService { export class OfflineService {
/** private reason: OfflineReason | null;
* BehaviorSubject to receive further status values.
*/
private offline = new BehaviorSubject<boolean>(false);
private bannerDefinition: BannerDefinition = {
text: _('Offline mode'),
icon: 'cloud_off'
};
public constructor(private banner: BannerService, translate: TranslateService) { public constructor(
translate.onLangChange.subscribe(() => { private OpenSlides: OpenSlidesService,
this.bannerDefinition.text = translate.instant(this.bannerDefinition.text); private offlineBroadcastService: OfflineBroadcastService,
}); private operatorService: OperatorService,
} private communicationManager: CommunicationManagerService
) {
/** this.offlineBroadcastService.goOfflineObservable.subscribe((reason: OfflineReason) => this.goOffline(reason));
* Determines of you are either in Offline mode or not connected via websocket
*
* @returns whether the client is offline or not connected
*/
public isOffline(): Observable<boolean> {
return this.offline;
}
/**
* Sets the offline flag. Restores the DataStoreService to the last known configuration.
*/
public goOfflineBecauseFailedWhoAmI(): void {
if (!this.offline.getValue()) {
console.log('offline because whoami failed.');
}
this.goOffline();
}
/**
* Sets the offline flag, because there is no connection to the server.
*/
public goOfflineBecauseConnectionLost(): void {
if (!this.offline.getValue()) {
console.log('offline because connection lost.');
}
this.goOffline();
} }
/** /**
* Helper function to set offline status * Helper function to set offline status
*/ */
private goOffline(): void { public goOffline(reason: OfflineReason): void {
this.offline.next(true); if (this.offlineBroadcastService.isOffline()) {
this.banner.addBanner(this.bannerDefinition); return;
}
this.reason = reason;
if (reason === OfflineReason.ConnectionLost) {
console.log('offline because connection lost.');
} else if (reason === OfflineReason.WhoAmIFailed) {
console.log('offline because whoami failed.');
} else {
console.error('No such offline reason', reason);
}
this.offlineBroadcastService.isOfflineSubject.next(true);
this.checkStillOffline();
}
private checkStillOffline(): void {
const timeout = Math.floor(Math.random() * 3000 + 2000);
console.log(`Try to go online in ${timeout} ms`);
setTimeout(async () => {
let online: boolean;
let whoami: WhoAmI | null = null;
if (this.reason === OfflineReason.ConnectionLost) {
online = await this.communicationManager.isCommunicationServiceOnline();
console.log('is communication online? ', online);
} else if (this.reason === OfflineReason.WhoAmIFailed) {
const result = await this.operatorService.whoAmI();
online = result.online;
whoami = result.whoami;
console.log('is whoami reachable?', online);
}
if (online) {
await this.goOnline(whoami);
// TODO: check all other reasons -> e.g. if the
// connection was lost, the operator must be checked and the other way
// around the comminucation must be started!!
// stop trying.
} else {
// continue trying.
this.checkStillOffline();
}
}, timeout);
} }
/** /**
* Function to return to online-status. * Function to return to online-status.
*
* First, we have to check, if all other sources (except this.reason) are online, too.
* This results in definetly having a whoami response at this point.
* If this is the case, we need to setup everything again:
* 1) check the operator. If this allowes for an logged in state (or anonymous is OK), do
* step 2, otherwise done.
* 2) enable communications.
*/ */
public goOnline(): void { private async goOnline(whoami?: WhoAmI): Promise<void> {
this.offline.next(false); console.log('go online!', this.reason, whoami);
this.banner.removeBanner(this.bannerDefinition); if (this.reason === OfflineReason.ConnectionLost) {
// now we have to check whoami
const result = await this.operatorService.whoAmI();
if (!result.online) {
console.log('whoami down.');
this.reason = OfflineReason.WhoAmIFailed;
this.checkStillOffline();
return;
}
whoami = result.whoami;
} else if (this.reason === OfflineReason.WhoAmIFailed) {
const online = await this.communicationManager.isCommunicationServiceOnline();
if (!online) {
console.log('communication down.');
this.reason = OfflineReason.ConnectionLost;
this.checkStillOffline();
return;
}
}
console.log('we are online!');
// Ok, we are online now!
const isLoggedIn = await this.OpenSlides.checkWhoAmI(whoami);
console.log('logged in:', isLoggedIn);
if (isLoggedIn) {
this.communicationManager.startCommunication();
}
console.log('done');
this.offlineBroadcastService.isOfflineSubject.next(false);
} }
} }

View File

@ -3,12 +3,11 @@ import { Router } from '@angular/router';
import { BehaviorSubject } from 'rxjs'; import { BehaviorSubject } from 'rxjs';
import { AutoupdateService } from './autoupdate.service'; import { CommunicationManagerService } from './communication-manager.service';
import { ConstantsService } from './constants.service';
import { DataStoreService } from './data-store.service'; import { DataStoreService } from './data-store.service';
import { OperatorService } from './operator.service'; import { OfflineBroadcastService, OfflineReason } from './offline-broadcast.service';
import { OperatorService, WhoAmI } from './operator.service';
import { StorageService } from './storage.service'; import { StorageService } from './storage.service';
import { WebsocketService } from './websocket.service';
/** /**
* Handles the bootup/showdown of this application. * Handles the bootup/showdown of this application.
@ -35,29 +34,19 @@ export class OpenSlidesService {
return this.booted.value; return this.booted.value;
} }
/**
* Constructor to create the OpenSlidesService. Registers itself to the WebsocketService.
* @param storageService
* @param operator
* @param websocketService
* @param router
* @param autoupdateService
* @param DS
*/
public constructor( public constructor(
private storageService: StorageService, private storageService: StorageService,
private operator: OperatorService, private operator: OperatorService,
private websocketService: WebsocketService,
private router: Router, private router: Router,
private autoupdateService: AutoupdateService,
private DS: DataStoreService, private DS: DataStoreService,
private constantsService: ConstantsService private communicationManager: CommunicationManagerService,
private offlineBroadcastService: OfflineBroadcastService
) { ) {
// Handler that gets called, if the websocket connection reconnects after a disconnection. // Handler that gets called, if the websocket connection reconnects after a disconnection.
// There might have changed something on the server, so we check the operator, if he changed. // There might have changed something on the server, so we check the operator, if he changed.
websocketService.retryReconnectEvent.subscribe(() => { /*websocketService.retryReconnectEvent.subscribe(() => {
this.checkOperator(); this.checkOperator();
}); });*/
this.bootup(); this.bootup();
} }
@ -68,20 +57,24 @@ export class OpenSlidesService {
*/ */
public async bootup(): Promise<void> { public async bootup(): Promise<void> {
// start autoupdate if the user is logged in: // start autoupdate if the user is logged in:
let response = await this.operator.whoAmIFromStorage(); let whoami = await this.operator.whoAmIFromStorage();
const needToCheckOperator = !!response; const needToCheckOperator = !!whoami;
if (!response) { if (!whoami) {
response = await this.operator.whoAmI(); const response = await this.operator.whoAmI();
if (!response.online) {
this.offlineBroadcastService.goOffline(OfflineReason.WhoAmIFailed);
}
whoami = response.whoami;
} }
if (!response.user && !response.guest_enabled) { if (!whoami.user && !whoami.guest_enabled) {
if (!location.pathname.includes('error')) { if (!location.pathname.includes('error')) {
this.redirectUrl = location.pathname; this.redirectUrl = location.pathname;
} }
this.redirectToLoginIfNotSubpage(); this.redirectToLoginIfNotSubpage();
} else { } else {
await this.afterLoginBootup(response.user_id); await this.afterLoginBootup(whoami.user_id);
} }
if (needToCheckOperator) { if (needToCheckOperator) {
@ -121,7 +114,7 @@ export class OpenSlidesService {
await this.DS.clear(); await this.DS.clear();
await this.storageService.set('lastUserLoggedIn', userId); await this.storageService.set('lastUserLoggedIn', userId);
} }
await this.setupDataStoreAndWebSocket(); await this.setupDataStoreAndStartCommunication();
// Now finally booted. // Now finally booted.
this.booted.next(true); this.booted.next(true);
} }
@ -129,23 +122,16 @@ export class OpenSlidesService {
/** /**
* Init DS from cache and after this start the websocket service. * Init DS from cache and after this start the websocket service.
*/ */
private async setupDataStoreAndWebSocket(): Promise<void> { private async setupDataStoreAndStartCommunication(): Promise<void> {
const changeId = await this.DS.initFromStorage(); await this.DS.initFromStorage();
// disconnect the WS connection, if there was one. This is needed this.communicationManager.startCommunication();
// to update the connection parameters, namely the cookies. If the user
// is changed, the WS needs to reconnect, so the new connection holds the new
// user information.
if (this.websocketService.isConnected) {
await this.websocketService.close(); // Wait for the disconnect.
}
await this.websocketService.connect(changeId); // Request changes after changeId.
} }
/** /**
* Shuts down OpenSlides. The websocket connection is closed and the operator is not set. * Shuts down OpenSlides.
*/ */
public async shutdown(): Promise<void> { public async shutdown(): Promise<void> {
await this.websocketService.close(); this.communicationManager.closeConnections();
this.booted.next(false); this.booted.next(false);
} }
@ -167,29 +153,37 @@ export class OpenSlidesService {
await this.bootup(); await this.bootup();
} }
public async checkOperator(requestChanges: boolean = true): Promise<void> {
const response = await this.operator.whoAmI();
if (!response.online) {
this.offlineBroadcastService.goOffline(OfflineReason.WhoAmIFailed);
}
await this.checkWhoAmI(response.whoami, requestChanges);
}
/** /**
* Verify that the operator is the same as it was before. Should be alled on a reconnect. * Verify that the operator is the same as it was before. Should be alled on a reconnect.
*
* @returns true, if the user is still logged in
*/ */
private async checkOperator(requestChanges: boolean = true): Promise<void> { public async checkWhoAmI(whoami: WhoAmI, requestChanges: boolean = true): Promise<boolean> {
const response = await this.operator.whoAmI(); let isLoggedIn = false;
// User logged off. // User logged off.
if (!response.user && !response.guest_enabled) { if (!whoami.user && !whoami.guest_enabled) {
this.websocketService.cancelReconnectenRetry();
await this.shutdown(); await this.shutdown();
this.redirectToLoginIfNotSubpage(); this.redirectToLoginIfNotSubpage();
} else { } else {
isLoggedIn = true;
if ( if (
(this.operator.user && this.operator.user.id !== response.user_id) || (this.operator.user && this.operator.user.id !== whoami.user_id) ||
(!this.operator.user && response.user_id) (!this.operator.user && whoami.user_id)
) { ) {
// user changed // user changed
await this.DS.clear(); await this.DS.clear();
await this.reboot(); await this.reboot();
} else if (requestChanges) {
// User is still the same, but check for missed autoupdates.
this.autoupdateService.requestChanges();
this.constantsService.refresh();
} }
} }
return isLoggedIn;
} }
} }

View File

@ -10,7 +10,6 @@ import { CollectionStringMapperService } from './collection-string-mapper.servic
import { DataStoreService } from './data-store.service'; import { DataStoreService } from './data-store.service';
import { Deferred } from '../promises/deferred'; import { Deferred } from '../promises/deferred';
import { HttpService } from './http.service'; import { HttpService } from './http.service';
import { OfflineService } from './offline.service';
import { OnAfterAppsLoaded } from '../definitions/on-after-apps-loaded'; import { OnAfterAppsLoaded } from '../definitions/on-after-apps-loaded';
import { OpenSlidesStatusService } from './openslides-status.service'; import { OpenSlidesStatusService } from './openslides-status.service';
import { StorageService } from './storage.service'; import { StorageService } from './storage.service';
@ -207,7 +206,6 @@ export class OperatorService implements OnAfterAppsLoaded {
public constructor( public constructor(
private http: HttpService, private http: HttpService,
private DS: DataStoreService, private DS: DataStoreService,
private offlineService: OfflineService,
private collectionStringMapper: CollectionStringMapperService, private collectionStringMapper: CollectionStringMapperService,
private storageService: StorageService, private storageService: StorageService,
private OSStatus: OpenSlidesStatusService private OSStatus: OpenSlidesStatusService
@ -306,18 +304,19 @@ export class OperatorService implements OnAfterAppsLoaded {
* *
* @returns The response of the WhoAmI request. * @returns The response of the WhoAmI request.
*/ */
public async whoAmI(): Promise<WhoAmI> { public async whoAmI(): Promise<{ whoami: WhoAmI; online: boolean }> {
let online = true;
try { try {
const response = await this.http.get(environment.urlPrefix + '/users/whoami/'); const response = await this.http.get(environment.urlPrefix + '/users/whoami/');
if (isWhoAmI(response)) { if (isWhoAmI(response)) {
await this.updateCurrentWhoAmI(response); await this.updateCurrentWhoAmI(response);
} else { } else {
this.offlineService.goOfflineBecauseFailedWhoAmI(); online = false;
} }
} catch (e) { } catch (e) {
this.offlineService.goOfflineBecauseFailedWhoAmI(); online = false;
} }
return this.currentWhoAmI; return { whoami: this.currentWhoAmI, online };
} }
/** /**

View File

@ -1,17 +0,0 @@
import { inject, TestBed } from '@angular/core/testing';
import { E2EImportsModule } from '../../../e2e-imports.module';
import { PingService } from './ping.service';
describe('PingService', () => {
beforeEach(() => {
TestBed.configureTestingModule({
imports: [E2EImportsModule],
providers: [PingService]
});
});
it('should be created', inject([PingService], (service: PingService) => {
expect(service).toBeTruthy();
}));
});

View File

@ -1,86 +0,0 @@
import { Injectable } from '@angular/core';
import { ConstantsService } from './constants.service';
import { Deferred } from '../promises/deferred';
import { TimeoutPromise } from '../promises/timeout-promise';
import { WebsocketService } from './websocket.service';
interface OpenSlidesSettings {
PING_INTERVAL?: number;
PING_TIMEOUT?: number;
}
@Injectable({
providedIn: 'root'
})
export class PingService {
/**
* The interval.
*/
private pingInterval: any;
private intervalTime = 30000;
private timeoutTime = 5000;
private lastLatency: number | null = null;
public constructor(private websocketService: WebsocketService, private constantsService: ConstantsService) {
this.setup();
}
private async setup(): Promise<void> {
const gotConstants = new Deferred();
this.constantsService.get<OpenSlidesSettings>('Settings').subscribe(settings => {
this.intervalTime = settings.PING_INTERVAL || 30000;
this.timeoutTime = settings.PING_TIMEOUT || 5000;
gotConstants.resolve();
});
await gotConstants;
// Connects the ping-pong mechanism to the opening and closing of the connection.
this.websocketService.closeEvent.subscribe(() => this.stopPing());
this.websocketService.generalConnectEvent.subscribe(() => this.startPing());
if (this.websocketService.isConnected) {
this.startPing();
}
}
/**
* Starts the ping-mechanism
*/
private startPing(): void {
if (this.pingInterval) {
return;
}
this.pingInterval = setInterval(async () => {
const start = performance.now();
try {
await TimeoutPromise(
this.websocketService.sendAndGetResponse('ping', this.lastLatency),
this.timeoutTime
);
this.lastLatency = performance.now() - start;
if (this.lastLatency > 1000) {
console.warn(`Ping took ${this.lastLatency / 1000} seconds.`);
}
} catch (e) {
console.warn(`The server didn't respond to ping within ${this.timeoutTime / 1000} seconds.`);
this.stopPing();
this.websocketService.simulateAbnormalClose();
}
}, this.intervalTime);
}
/**
* Clears the ping interval
*/
private stopPing(): void {
if (this.pingInterval) {
clearInterval(this.pingInterval);
this.pingInterval = null;
}
}
}

View File

@ -1,17 +0,0 @@
import { inject, TestBed } from '@angular/core/testing';
import { E2EImportsModule } from '../../../e2e-imports.module';
import { PrioritizeService } from './prioritize.service';
describe('PrioritizeService', () => {
beforeEach(() => {
TestBed.configureTestingModule({
imports: [E2EImportsModule],
providers: [PrioritizeService]
});
});
it('should be created', inject([PrioritizeService], (service: PrioritizeService) => {
expect(service).toBeTruthy();
}));
});

View File

@ -1,46 +0,0 @@
import { Injectable } from '@angular/core';
import { ConstantsService } from './constants.service';
import { DataStoreService } from './data-store.service';
import { OpenSlidesStatusService } from './openslides-status.service';
import { OperatorService } from './operator.service';
import { WebsocketService } from './websocket.service';
interface OpenSlidesSettings {
PRIORITIZED_GROUP_IDS?: number[];
}
/**
* Cares about prioritizing a client. Checks, if the operator is in one of
* some prioritized groups. These group ids come from the server. If the prio-
* ritization changes, the websocket connection gets reconnected.
*/
@Injectable({
providedIn: 'root'
})
export class PrioritizeService {
private prioritizedGroupIds: number[] = [];
public constructor(
constantsService: ConstantsService,
private websocketService: WebsocketService,
private DS: DataStoreService,
private openSlidesStatusService: OpenSlidesStatusService,
private operator: OperatorService
) {
constantsService.get<OpenSlidesSettings>('Settings').subscribe(settings => {
this.prioritizedGroupIds = settings.PRIORITIZED_GROUP_IDS || [];
this.checkPrioritization();
});
operator.getUserObservable().subscribe(() => this.checkPrioritization());
}
private checkPrioritization(): void {
const opPrioritized = this.operator.isInGroupIdsNonAdminCheck(...this.prioritizedGroupIds);
if (this.openSlidesStatusService.isPrioritizedClient !== opPrioritized) {
console.log('Alter prioritization:', opPrioritized);
this.openSlidesStatusService.isPrioritizedClient = opPrioritized;
this.websocketService.reconnect(this.DS.maxChangeId);
}
}
}

View File

@ -3,8 +3,8 @@ import { Injectable } from '@angular/core';
import { BehaviorSubject, Observable, Subject } from 'rxjs'; import { BehaviorSubject, Observable, Subject } from 'rxjs';
import { auditTime } from 'rxjs/operators'; import { auditTime } from 'rxjs/operators';
import { WebsocketService } from 'app/core/core-services/websocket.service';
import { Projector, ProjectorElement } from 'app/shared/models/core/projector'; import { Projector, ProjectorElement } from 'app/shared/models/core/projector';
import { CommunicationManagerService, OfflineError } from './communication-manager.service';
export interface SlideData<T = { error?: string }, P extends ProjectorElement = ProjectorElement> { export interface SlideData<T = { error?: string }, P extends ProjectorElement = ProjectorElement> {
data: T; data: T;
@ -15,13 +15,13 @@ export interface SlideData<T = { error?: string }, P extends ProjectorElement =
export type ProjectorData = SlideData[]; export type ProjectorData = SlideData[];
interface AllProjectorData { interface AllProjectorData {
[id: number]: ProjectorData | { error: string }; [id: number]: ProjectorData;
} }
/** /**
* Received data from server. * Received data from server.
*/ */
interface ProjectorWebsocketMessage { interface ProjectorDataMessage {
/** /**
* The `change_id` of the current update. * The `change_id` of the current update.
*/ */
@ -63,38 +63,63 @@ export class ProjectorDataService {
*/ */
private currentChangeId = 0; private currentChangeId = 0;
/** private streamCloseFn: () => void | null = null;
* Constructor.
*
* @param websocketService
*/
public constructor(private websocketService: WebsocketService) {
// Dispatch projector data.
this.websocketService.getOberservable('projector').subscribe((update: ProjectorWebsocketMessage) => {
if (this.currentChangeId > update.change_id) {
return;
}
Object.keys(update.data).forEach(_id => {
const id = parseInt(_id, 10);
if (this.currentProjectorData[id]) {
this.currentProjectorData[id].next(update.data[id] as ProjectorData);
}
});
this.currentChangeId = update.change_id;
});
// The service need to re-register, if the websocket connection was lost. public constructor(private communicationManager: CommunicationManagerService) {
this.websocketService.generalConnectEvent.subscribe(() => this.updateProjectorDataSubscription()); this.communicationManager.startCommunicationEvent.subscribe(() => this.updateProjectorDataSubscription());
// With a bit of debounce, update the needed projectors. // With a bit of debounce, update the needed projectors.
this.updateProjectorDataDebounceSubject.pipe(auditTime(10)).subscribe(() => { this.updateProjectorDataDebounceSubject.pipe(auditTime(10)).subscribe(() => {
const allActiveProjectorIds = Object.keys(this.openProjectorInstances) const allActiveProjectorIds = Object.keys(this.openProjectorInstances)
.map(id => parseInt(id, 10)) .map(id => parseInt(id, 10))
.filter(id => this.openProjectorInstances[id] > 0); .filter(id => this.openProjectorInstances[id] > 0);
this.websocketService.send('listenToProjectors', { projector_ids: allActiveProjectorIds }); this.requestProjectors(allActiveProjectorIds);
}); });
} }
public async requestProjectors(allActiveProjectorIds: number[]): Promise<void> {
this.cancelCurrentServerSubscription();
if (allActiveProjectorIds.length === 0) {
return;
}
try {
this.streamCloseFn = await this.communicationManager.subscribe<ProjectorDataMessage>(
'/system/projector',
message => {
this.handleMesage(message);
},
() => ({ projector_ids: allActiveProjectorIds.join(',') })
);
} catch (e) {
if (!(e instanceof OfflineError)) {
console.log(e);
}
}
}
public cancelCurrentServerSubscription(): void {
if (this.streamCloseFn) {
this.streamCloseFn();
this.streamCloseFn = null;
}
}
private handleMesage(message: ProjectorDataMessage): void {
if (this.currentChangeId > message.change_id) {
console.log('Projector: Change id too low:', this.currentChangeId, message.change_id);
return;
}
Object.keys(message.data).forEach(_id => {
const id = parseInt(_id, 10);
if (this.currentProjectorData[id]) {
this.currentProjectorData[id].next(message.data[id] as ProjectorData);
}
});
this.currentChangeId = message.change_id;
}
/** /**
* Gets an observable for the projector data. * Gets an observable for the projector data.
* *

View File

@ -40,12 +40,6 @@ export interface ProjectorTitle {
providedIn: 'root' providedIn: 'root'
}) })
export class ProjectorService { export class ProjectorService {
/**
* Constructor.
*
* @param DS
* @param dataSend
*/
public constructor( public constructor(
private DS: DataStoreService, private DS: DataStoreService,
private http: HttpService, private http: HttpService,

View File

@ -0,0 +1,281 @@
import {
HttpClient,
HttpDownloadProgressEvent,
HttpEvent,
HttpHeaderResponse,
HttpHeaders,
HttpParams
} from '@angular/common/http';
import { Injectable } from '@angular/core';
import { Observable, Subscription } from 'rxjs';
import { HTTPMethod } from '../definitions/http-methods';
const HEADER_EVENT_TYPE = 2;
const PROGRESS_EVENT_TYPE = 3;
const FINISH_EVENT_TYPE = 4;
export type Params = HttpParams | { [param: string]: string | string[] };
export enum ErrorType {
Client,
Server, // or network errors, they are the same.
Unknown
}
export function verboseErrorType(type: ErrorType): string {
switch (type) {
case ErrorType.Client:
return 'Client';
case ErrorType.Server:
return 'Server';
case ErrorType.Unknown:
return 'Unknown';
}
}
export interface CommunicationError {
type: string;
msg: string;
}
export function isCommunicationError(obj: any): obj is CommunicationError {
const _obj = obj as CommunicationError;
return typeof obj === 'object' && typeof _obj.msg === 'string' && typeof _obj.type === 'string';
}
interface CommunicationErrorWrapper {
error: CommunicationError;
}
export function isCommunicationErrorWrapper(obj: any): obj is CommunicationErrorWrapper {
return typeof obj === 'object' && isCommunicationError(obj.error);
}
export type ErrorHandler = (type: ErrorType, error: CommunicationError, message: string) => void;
export class StreamConnectionError extends Error {
public constructor(public code: number, message: string) {
super(message);
this.name = 'StreamConnectionError';
}
}
export class StreamContainer {
public readonly id = Math.floor(Math.random() * (900000 - 1) + 100000); // [100000, 999999]
public hasErroredAmount = 0;
public stream?: Stream<any>;
public constructor(
public url: string,
public messageHandler: (message: any) => void,
public params: () => Params
) {}
}
export class Stream<T> {
private subscription: Subscription = null;
private hasError = false;
private reportedError = false;
private _statuscode: number;
public get statuscode(): number {
return this._statuscode;
}
private _errorContent: CommunicationError;
public get errorContent(): CommunicationError {
return this._errorContent;
}
/**
* This is the index where we checked, if there is a \n in the read buffer (event.partialText)
* This position is always >= contentStartIndex and is > contentStartIndex, if the message
* was too big to fit into one buffer. So we have just a partial message.
*
* The difference between this index and contentStartIndex is that this index remembers the position
* we checked for a \n which lay in the middle of the next JOSN-packet.
*/
private checkedUntilIndex = 0;
/**
* This index holds always the position of the current JOSN-packet, that we are receiving.
*/
private contentStartIndex = 0;
private closed = false;
public constructor(
observable: Observable<HttpEvent<string>>,
private messageHandler: (message: T) => void,
private errorHandler: ErrorHandler
) {
this.subscription = observable.subscribe(
(event: HttpEvent<string>) => {
if (this.closed) {
return;
}
if (event.type === HEADER_EVENT_TYPE) {
const headerResponse = event as HttpHeaderResponse;
this._statuscode = headerResponse.status;
if (headerResponse.status >= 400) {
this.hasError = true;
}
} else if ((<HttpEvent<string>>event).type === PROGRESS_EVENT_TYPE) {
this.handleMessage(event as HttpDownloadProgressEvent);
} else if ((<HttpEvent<string>>event).type === FINISH_EVENT_TYPE) {
this.errorHandler(ErrorType.Server, null, 'The stream was closed');
}
},
error => {
this.errorHandler(ErrorType.Server, error, 'Network error');
},
() => {
console.log('The stream was completed');
}
);
}
private handleMessage(event: HttpDownloadProgressEvent): void {
if (this.hasError) {
if (!this.reportedError) {
this.reportedError = true;
// try to get the `error` key from object
this._errorContent = this.tryParseError(event.partialText);
this.errorHandler(this.getErrorTypeFromStatusCode(), this._errorContent, 'Reported error by server');
}
return;
}
// Maybe we get multiple messages, so continue, until the complete buffer is checked.
while (this.checkedUntilIndex < event.loaded) {
// check if there is a \n somewhere in [checkedUntilIndex, ...]
const LF_index = event.partialText.indexOf('\n', this.checkedUntilIndex);
if (LF_index >= 0) {
// take string in [contentStartIndex, LF_index-1]. This must be valid JSON.
// In substring, the last character is exlusive.
const content = event.partialText.substring(this.contentStartIndex, LF_index);
// move pointer: next JSON starts at LF_index + 1
this.checkedUntilIndex = LF_index + 1;
this.contentStartIndex = LF_index + 1;
const parsedContent = this.tryParseJson(content);
if (isCommunicationError(parsedContent)) {
if (this.hasError && this.reportedError) {
return;
}
this.hasError = true;
this._errorContent = parsedContent;
// Do not trigger the error handler, if the connection-retry-routine is still handling this issue
if (!this.reportedError) {
this.reportedError = true;
console.error(this._errorContent);
this.errorHandler(
this.getErrorTypeFromStatusCode(),
this._errorContent,
'Reported error by server'
);
}
return;
} else {
// check, if we didn't get a keep alive
if (Object.keys(parsedContent).length > 0) {
console.debug('received', parsedContent);
this.messageHandler(parsedContent);
}
}
} else {
this.checkedUntilIndex = event.loaded;
}
}
}
private getErrorTypeFromStatusCode(): ErrorType {
if (!this.statuscode) {
return ErrorType.Unknown;
}
if (this.statuscode >= 400 && this.statuscode < 500) {
return ErrorType.Client;
}
if (this.statuscode >= 500) {
return ErrorType.Server;
}
return ErrorType.Unknown;
}
private tryParseJson(json: string): T | CommunicationError {
try {
return JSON.parse(json) as T;
} catch (e) {
return this.tryParseError(json);
}
}
/**
* This one is a bit tricky. Error can be:
* - string with HTML, e.g. provided by proxies if the service is unreachable
* - string with json of form {"error": {"type": ..., "msg": ...}}
*/
private tryParseError(error: any): CommunicationError {
if (typeof error === 'string') {
try {
error = JSON.parse(error);
} catch (e) {
return { type: 'Unknown Error', msg: error };
}
}
if (isCommunicationErrorWrapper(error)) {
return error.error;
} else if (isCommunicationError(error)) {
return error;
}
// we have something else.... ??
console.error('Unknown error', error);
throw new Error('Unknown error: ' + error.toString());
}
public close(): void {
this.subscription.unsubscribe();
this.subscription = null;
this.closed = true;
}
}
@Injectable({
providedIn: 'root'
})
export class StreamingCommunicationService {
public constructor(private http: HttpClient) {}
public subscribe<T>(streamContainer: StreamContainer, errorHandler: ErrorHandler): void {
const options: {
body?: any;
headers?: HttpHeaders | { [header: string]: string | string[] };
observe: 'events';
params?: HttpParams | { [param: string]: string | string[] };
reportProgress?: boolean;
responseType: 'text';
withCredentials?: boolean;
} = {
headers: {
'Content-Type': 'application/json',
'ngsw-bypass': 'yes'
},
responseType: 'text',
observe: 'events',
reportProgress: true
};
const params = streamContainer.params();
if (params) {
options.params = params;
}
const observable = this.http.request(HTTPMethod.GET, streamContainer.url, options);
if (streamContainer.stream) {
console.error('Illegal state!');
}
const stream = new Stream<T>(observable, streamContainer.messageHandler, errorHandler);
streamContainer.stream = stream;
}
}

View File

@ -9,7 +9,6 @@ import { DataStoreService, DataStoreUpdateManagerService } from './data-store.se
import { HttpService } from './http.service'; import { HttpService } from './http.service';
import { OpenSlidesStatusService } from './openslides-status.service'; import { OpenSlidesStatusService } from './openslides-status.service';
import { OpenSlidesService } from './openslides.service'; import { OpenSlidesService } from './openslides.service';
import { WebsocketService } from './websocket.service';
interface HistoryData { interface HistoryData {
[collection: string]: BaseModel[]; [collection: string]: BaseModel[];
@ -31,7 +30,6 @@ export class TimeTravelService {
* Constructs the time travel service * Constructs the time travel service
* *
* @param httpService To fetch the history data * @param httpService To fetch the history data
* @param webSocketService to disable websocket connection
* @param modelMapperService to cast history objects into models * @param modelMapperService to cast history objects into models
* @param DS to overwrite the dataStore * @param DS to overwrite the dataStore
* @param OSStatus Sets the history status * @param OSStatus Sets the history status
@ -39,7 +37,6 @@ export class TimeTravelService {
*/ */
public constructor( public constructor(
private httpService: HttpService, private httpService: HttpService,
private webSocketService: WebsocketService,
private modelMapperService: CollectionStringMapperService, private modelMapperService: CollectionStringMapperService,
private DS: DataStoreService, private DS: DataStoreService,
private OSStatus: OpenSlidesStatusService, private OSStatus: OpenSlidesStatusService,
@ -100,7 +97,8 @@ export class TimeTravelService {
* Clears the DataStore and stops the WebSocket connection * Clears the DataStore and stops the WebSocket connection
*/ */
private async stopTime(history: History): Promise<void> { private async stopTime(history: History): Promise<void> {
await this.webSocketService.close(); // await this.webSocketService.close();
// TODO
await this.DS.set(); // Same as clear, but not persistent. await this.DS.set(); // Same as clear, but not persistent.
this.OSStatus.enterHistoryMode(history); this.OSStatus.enterHistoryMode(history);
} }

View File

@ -1,17 +0,0 @@
import { inject, TestBed } from '@angular/core/testing';
import { E2EImportsModule } from '../../../e2e-imports.module';
import { WebsocketService } from './websocket.service';
describe('WebsocketService', () => {
beforeEach(() => {
TestBed.configureTestingModule({
imports: [E2EImportsModule],
providers: [WebsocketService]
});
});
it('should be created', inject([WebsocketService], (service: WebsocketService) => {
expect(service).toBeTruthy();
}));
});

View File

@ -1,551 +0,0 @@
import { EventEmitter, Injectable, NgZone } from '@angular/core';
import { MatSnackBarRef, SimpleSnackBar } from '@angular/material/snack-bar';
import { Router } from '@angular/router';
import { compress, decompress } from 'lz4js';
import { Observable, Subject } from 'rxjs';
import { take } from 'rxjs/operators';
import { OfflineService } from './offline.service';
import { OpenSlidesStatusService } from './openslides-status.service';
import { formatQueryParams, QueryParams } from '../definitions/query-params';
/**
* The generic message format in which messages are send and recieved by the server.
*/
interface BaseWebsocketMessage {
type: string;
content: any;
}
/**
* Outgoing messages must have an id.
*/
interface OutgoingWebsocketMessage extends BaseWebsocketMessage {
id: string;
}
/**
* Incomming messages may have an `in_response`, if they are an answer to a previously
* submitted request.
*/
interface IncommingWebsocketMessage extends BaseWebsocketMessage {
in_response?: string;
}
/**
* The format of a messages content, if the message type is "error"
*/
interface WebsocketErrorContent {
code: number;
message: string;
}
function isWebsocketErrorContent(obj: any): obj is WebsocketErrorContent {
return !!obj && obj.code !== undefined && obj.message !== undefined;
}
/**
* All (custom) error codes that are used to pass error information
* from the server to the client
*/
export const WEBSOCKET_ERROR_CODES = {
NOT_AUTHORIZED: 100,
CHANGE_ID_TOO_HIGH: 101,
WRONG_FORMAT: 102
};
/**
* Service that handles WebSocket connections. Other services can register themselfs
* with {@method getOberservable} for a specific type of messages. The content will be published.
*/
@Injectable({
providedIn: 'root'
})
export class WebsocketService {
/**
* The reference to the snackbar entry that is shown, if the connection is lost.
*/
private connectionErrorNotice: MatSnackBarRef<SimpleSnackBar>;
/**
* Subjects that will be called, if a reconnect after a retry (e.g. with a previous
* connection loss) was successful.
*/
private readonly _retryReconnectEvent: EventEmitter<void> = new EventEmitter<void>();
/**
* Getter for the retry reconnect event.
*/
public get retryReconnectEvent(): EventEmitter<void> {
return this._retryReconnectEvent;
}
/**
* Subjects that will be called, if connect took place, but not a retry reconnect.
* THis is the complement from the generalConnectEvent to the retryReconnectEvent.
*/
private readonly _noRetryConnectEvent: EventEmitter<void> = new EventEmitter<void>();
/**
* Getter for the no-retry connect event.
*/
public get noRetryConnectEvent(): EventEmitter<void> {
return this._noRetryConnectEvent;
}
/**
* Listeners will be nofitied, if the wesocket connection is establiched.
*/
private readonly _generalConnectEvent: EventEmitter<void> = new EventEmitter<void>();
/**
* Getter for the connect event.
*/
public get generalConnectEvent(): EventEmitter<void> {
return this._generalConnectEvent;
}
/**
* Listeners will be nofitied, if the wesocket connection is closed.
*/
private readonly _closeEvent: EventEmitter<void> = new EventEmitter<void>();
/**
* Getter for the close event.
*/
public get closeEvent(): EventEmitter<void> {
return this._closeEvent;
}
/**
* The subject for all websocket *message* errors (no connection errors).
*/
private readonly _errorResponseSubject = new Subject<WebsocketErrorContent>();
/**
* The error response obersable for all websocket message errors.
*/
public get errorResponseObservable(): Observable<WebsocketErrorContent> {
return this._errorResponseSubject.asObservable();
}
/**
* Saves, if the connection is open
*/
private _connectionOpen = false;
/**
* Whether the WebSocket connection is established
*/
public get isConnected(): boolean {
return this._connectionOpen;
}
private sendQueueWhileNotConnected: (string | ArrayBuffer)[] = [];
/**
* The websocket.
*/
private websocket: WebSocket | null;
private websocketId: string | null;
/**
* Subjects for types of websocket messages. A subscriber can get an Observable by {@function getOberservable}.
*/
private subjects: { [type: string]: Subject<any> } = {};
/**
* Callbacks for a waiting response. If any callback returns true, the message/error will not be propagated with the
* responsible subjects for the message type.
*/
private responseCallbacks: {
[id: string]: [(val: any) => boolean, (error: WebsocketErrorContent) => boolean];
} = {};
/**
* Saves, if the WS Connection should be closed (e.g. after an explicit `close()`). Prohibits
* retry connection attempts.
*/
private shouldBeClosed = true;
/**
* Counter for delaying the offline message.
*/
private retryCounter = 0;
/**
* The timeout in the onClose-handler for the next reconnect retry.
*/
private retryTimeout: any = null;
/**
* Constructor that handles the router
*
* @param zone
* @param router
* @param openSlidesStatusService
* @param offlineService
*/
public constructor(
private zone: NgZone,
private router: Router,
private openSlidesStatusService: OpenSlidesStatusService,
private offlineService: OfflineService
) {}
/**
* Creates a new WebSocket connection and handles incomming events.
*
* Uses NgZone to let all callbacks run in the angular context.
*/
public async connect(changeId: number | null = null, retry: boolean = false): Promise<void> {
const websocketId = Math.random().toString(36).substring(7);
this.websocketId = websocketId;
if (this.websocket) {
this.websocket.close();
this.websocket = null;
}
if (!retry) {
this.shouldBeClosed = false;
}
const queryParams: QueryParams = {};
if (changeId !== null) {
queryParams.change_id = changeId;
}
// Create the websocket
let socketPath = location.protocol === 'https:' ? 'wss://' : 'ws://';
socketPath += window.location.host;
if (this.openSlidesStatusService.isPrioritizedClient) {
socketPath += '/prioritize';
}
socketPath += '/ws/';
socketPath += formatQueryParams(queryParams);
this.websocket = new WebSocket(socketPath);
this.websocket.binaryType = 'arraybuffer';
// connection established. If this connect attept was a retry,
// The error notice will be removed and the reconnectSubject is published.
this.websocket.onopen = (event: Event) => {
if (this.websocketId !== websocketId) {
return;
}
this.zone.run(() => {
this.retryCounter = 0;
if (this.shouldBeClosed) {
this.offlineService.goOnline();
return;
}
this._connectionOpen = true;
if (retry) {
this.offlineService.goOnline();
this._retryReconnectEvent.emit();
} else {
this._noRetryConnectEvent.emit();
}
this._generalConnectEvent.emit();
this.sendQueueWhileNotConnected.forEach(entry => {
this.websocket.send(entry);
});
this.sendQueueWhileNotConnected = [];
});
};
this.websocket.onmessage = (event: MessageEvent) => {
if (this.websocketId !== websocketId) {
return;
}
this.zone.run(() => {
this.handleMessage(event.data);
});
};
this.websocket.onclose = (event: CloseEvent) => {
if (this.websocketId !== websocketId) {
return;
}
this.zone.run(() => {
this.onclose();
});
};
this.websocket.onerror = (event: ErrorEvent) => {
if (this.websocketId !== websocketId) {
return;
}
// place for proper error handling and debugging.
// Required to get more information about errors
this.zone.run(() => {
console.warn('WS error event:', event);
});
};
}
/**
* Handles an incomming message.
*
* @param data The message
*/
private handleMessage(data: string | ArrayBuffer): void {
if (data instanceof ArrayBuffer) {
const compressedSize = data.byteLength;
const decompressedBuffer: Uint8Array = decompress(new Uint8Array(data));
console.debug(
`Recieved ${compressedSize / 1024} KB (${
decompressedBuffer.byteLength / 1024
} KB uncompressed), ratio ${decompressedBuffer.byteLength / compressedSize}`
);
data = this.arrayBufferToString(decompressedBuffer);
}
const message: IncommingWebsocketMessage = JSON.parse(data);
console.debug('Received', message);
const type = message.type;
const inResponse = message.in_response;
const callbacks = this.responseCallbacks[inResponse];
if (callbacks) {
delete this.responseCallbacks[inResponse];
}
if (type === 'error') {
if (!isWebsocketErrorContent(message.content)) {
console.error('Websocket error without standard form!', message);
return;
}
// Print this to the console.
const error = message.content;
const errorDescription =
Object.keys(WEBSOCKET_ERROR_CODES).find(key => WEBSOCKET_ERROR_CODES[key] === error.code) ||
'unknown code';
console.error(`Websocket error with code=${error.code} (${errorDescription}):`, error.message);
// call the error callback, if there is any. If it returns true (means "handled"),
// the errorResponseSubject will not be called
if (inResponse && callbacks && callbacks[1] && callbacks[1](error)) {
return;
}
this._errorResponseSubject.next(error);
return;
}
// Try to fire a response callback directly. If it returnes true, the message is handeled
// and not distributed further
if (inResponse && callbacks && callbacks[0](message.content)) {
return;
}
if (this.subjects[type]) {
// Pass the content to the registered subscribers.
this.subjects[type].next(message.content);
} else {
console.warn(
`Got unknown websocket message type "${type}" (inResponse: ${inResponse}) with content`,
message.content
);
}
}
/**
* Closes the connection error notice
*/
private onclose(): void {
if (this.websocket) {
this.websocketId = null; // set to null, so now further events will be
// registered with the line below.
this.websocket.close(); // Cleanup old connection
this.websocket = null;
}
this._connectionOpen = false;
// 1000 is a normal close, like the close on logout
this._closeEvent.emit();
if (!this.shouldBeClosed) {
// Do not show the message snackbar on the projector
// tests for /projector and /projector/<id>
const onProjector = this.router.url.match(/^\/projector(\/[0-9]+\/?)?$/);
if (this.retryCounter <= 3) {
this.retryCounter++;
}
if (!this.connectionErrorNotice && !onProjector && this.retryCounter > 3) {
this.offlineService.goOfflineBecauseConnectionLost();
}
// A random retry timeout between 2000 and 5000 ms.
const timeout = Math.floor(Math.random() * 3000 + 2000);
this.retryTimeout = setTimeout(() => {
this.retryTimeout = null;
this.connect(null, true);
}, timeout);
}
}
public cancelReconnectenRetry(): void {
if (this.retryTimeout) {
clearTimeout(this.retryTimeout);
this.retryTimeout = null;
}
}
/**
* Closes the websocket connection.
*/
public async close(): Promise<void> {
this.shouldBeClosed = true;
this.offlineService.goOnline();
if (this.websocket) {
this.websocket.close();
this.websocket = null;
await this.closeEvent.pipe(take(1)).toPromise();
}
}
/**
* Simulates an abnormal close.
*
* Internally does not set `shouldBeClosed`, so a reconnect is forced.
*/
public simulateAbnormalClose(): void {
this.onclose();
}
/**
* closes and reopens the connection. If the connection was closed before,
* it will be just opened.
*
* @param options The options for the new connection
*/
public async reconnect(changeId: number | null = null): Promise<void> {
await this.close();
await this.connect(changeId);
}
/**
* Returns an observable for messages of the given type.
* @param type the message type
*/
public getOberservable<T>(type: string): Observable<T> {
if (!this.subjects[type]) {
this.subjects[type] = new Subject<T>();
}
return this.subjects[type].asObservable();
}
/**
* Sends a message to the server with the content and the given type.
*
* @param type the message type
* @param content the actual content
* @param success an optional success callback for a response. If it returns true, the message will not be
* propagated through the recieve subjects.
* @param error an optional error callback for a response. If it returns true, the error will not be propagated
* with the error subject.
* @param id an optional id for the message. If not given, a random id will be generated and returned.
* @returns the message id
*/
public send<T, R>(
type: string,
content: T,
success?: (val: R) => boolean,
error?: (error: WebsocketErrorContent) => boolean,
id?: string
): string {
if (!this.websocket) {
return;
}
const message: OutgoingWebsocketMessage = {
type: type,
content: content,
id: id
};
// create message id if not given. Required by the server.
if (!message.id) {
message.id = '';
const possible = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
for (let i = 0; i < 8; i++) {
message.id += possible.charAt(Math.floor(Math.random() * possible.length));
}
}
if (success) {
this.responseCallbacks[message.id] = [success, error];
}
const jsonMessage = JSON.stringify(message);
const bytesMessage = this.stringToBuffer(jsonMessage);
const compressedMessage: ArrayBuffer = compress(bytesMessage);
const ratio = bytesMessage.byteLength / compressedMessage.byteLength;
const toSend = ratio > 1 ? compressedMessage : jsonMessage;
if (this.isConnected) {
this.websocket.send(toSend);
} else {
this.sendQueueWhileNotConnected.push(toSend);
}
return message.id;
}
/**
* Sends a message and waits for the response
*
* @param type the message type
* @param content the actual content
* @param id an optional id for the message. If not given, a random id will be generated and returned.
*/
public sendAndGetResponse<T, R>(type: string, content: T, id?: string): Promise<R> {
return new Promise<R>((resolve, reject) => {
this.send<T, R>(
type,
content,
val => {
resolve(val);
return true;
},
val => {
reject(val);
return true;
},
id
);
});
}
/**
* Converts an ArrayBuffer to a String.
*
* @param buffer - Buffer to convert
* @returns String
*/
private arrayBufferToString(buffer: Uint8Array): string {
return Array.from(buffer)
.map(code => String.fromCharCode(code))
.join('');
}
/**
* Converts a String to an ArrayBuffer.
*
* @param str - String to convert.
* @returns bufferView.
*/
private stringToBuffer(str: string): Uint8Array {
const bufferView = new Uint8Array();
for (let i = 0; i < str.length; i++) {
bufferView[i] = str.charCodeAt(i);
}
return bufferView;
}
}

View File

@ -0,0 +1,43 @@
export interface AutoupdateFormat {
/**
* All changed (and created) items as their full/restricted data grouped by their collection.
*/
changed: {
[collectionString: string]: object[];
};
/**
* All deleted items (by id) grouped by their collection.
*/
deleted: {
[collectionString: string]: number[];
};
/**
* The lower change id bond for this autoupdate
*/
from_change_id: number;
/**
* The upper change id bound for this autoupdate
*/
to_change_id: number;
/**
* Flag, if this autoupdate contains all data. If so, the DS needs to be resetted.
*/
all_data: boolean;
}
export function isAutoupdateFormat(obj: any): obj is AutoupdateFormat {
const format = obj as AutoupdateFormat;
return (
obj &&
typeof obj === 'object' &&
format.changed !== undefined &&
format.deleted !== undefined &&
format.from_change_id !== undefined &&
format.to_change_id !== undefined &&
format.all_data !== undefined
);
}

View File

@ -0,0 +1,10 @@
/**
* Enum for different HTTPMethods
*/
export enum HTTPMethod {
GET = 'get',
POST = 'post',
PUT = 'put',
PATCH = 'patch',
DELETE = 'delete'
}

View File

@ -14,16 +14,16 @@
* ``` * ```
*/ */
export class Deferred<T = void> extends Promise<T> { export class Deferred<T = void> extends Promise<T> {
/**
* The promise to wait for
*/
public readonly promise: Promise<T>;
/** /**
* custom resolve function * custom resolve function
*/ */
private _resolve: (val?: T) => void; private _resolve: (val?: T) => void;
private _wasResolved;
public get wasResolved(): boolean {
return this._wasResolved;
}
/** /**
* Creates the promise and overloads the resolve function * Creates the promise and overloads the resolve function
*/ */
@ -32,6 +32,7 @@ export class Deferred<T = void> extends Promise<T> {
super(resolve => { super(resolve => {
preResolve = resolve; preResolve = resolve;
}); });
this._wasResolved = false;
this._resolve = preResolve; this._resolve = preResolve;
} }
@ -40,5 +41,6 @@ export class Deferred<T = void> extends Promise<T> {
*/ */
public resolve(val?: T): void { public resolve(val?: T): void {
this._resolve(val); this._resolve(val);
this._wasResolved = true;
} }
} }

View File

@ -0,0 +1,10 @@
/**
* Wraps a promise and let it reject after the given timeout (in ms), if it was
* not resolved before this timeout.
*
* @param delay The time to sleep in miliseconds
* @returns a new Promise
*/
export function SleepPromise(delay: number): Promise<void> {
return new Promise((resolve, _) => setTimeout(resolve, delay));
}

View File

@ -0,0 +1,6 @@
import { MonoTypeOperatorFunction } from 'rxjs';
import { throttleTime } from 'rxjs/operators';
export function trailingThrottleTime<T = unknown>(time: number): MonoTypeOperatorFunction<T> {
return throttleTime<T>(time, undefined, { leading: false, trailing: true });
}

View File

@ -1,7 +1,10 @@
import { Injectable } from '@angular/core'; import { Injectable } from '@angular/core';
import { marker as _ } from '@biesbjerg/ngx-translate-extract-marker';
import { BehaviorSubject } from 'rxjs'; import { BehaviorSubject } from 'rxjs';
import { OfflineBroadcastService } from '../core-services/offline-broadcast.service';
export interface BannerDefinition { export interface BannerDefinition {
type?: string; type?: string;
class?: string; class?: string;
@ -20,8 +23,26 @@ export interface BannerDefinition {
providedIn: 'root' providedIn: 'root'
}) })
export class BannerService { export class BannerService {
private offlineBannerDefinition: BannerDefinition = {
text: _('Offline mode'),
icon: 'cloud_off'
};
public activeBanners: BehaviorSubject<BannerDefinition[]> = new BehaviorSubject<BannerDefinition[]>([]); public activeBanners: BehaviorSubject<BannerDefinition[]> = new BehaviorSubject<BannerDefinition[]>([]);
public constructor(/*translate: TranslateService, */ offlineBroadcastService: OfflineBroadcastService) {
/*translate.onLangChange.subscribe(() => {
this.offlineBannerDefinition.text = translate.instant(this.offlineBannerDefinition.text);
});*/
offlineBroadcastService.isOfflineObservable.subscribe(offline => {
if (offline) {
this.addBanner(this.offlineBannerDefinition);
} else {
this.removeBanner(this.offlineBannerDefinition);
}
});
}
/** /**
* Adds a banner to the list of active banners. Skip the banner if it's already in the list * Adds a banner to the list of active banners. Skip the banner if it's already in the list
* @param toAdd the banner to add * @param toAdd the banner to add

View File

@ -42,24 +42,24 @@ export class CountUsersService {
public constructor(private notifyService: NotifyService, operator: OperatorService) { public constructor(private notifyService: NotifyService, operator: OperatorService) {
// Listen for requests to send an answer. // Listen for requests to send an answer.
this.notifyService.getMessageObservable<CountUserRequest>(REQUEST_NAME).subscribe(request => { this.notifyService.getMessageObservable<CountUserRequest>(REQUEST_NAME).subscribe(request => {
if (request.content.token) { if (request.message.token) {
this.notifyService.sendToChannels<CountUserResponse>( this.notifyService.sendToChannels<CountUserResponse>(
RESPONSE_NAME, RESPONSE_NAME,
{ {
token: request.content.token, token: request.message.token,
data: { data: {
userId: this.currentUserId userId: this.currentUserId
} }
}, },
request.senderChannelName request.sender_channel_id
); );
} }
}); });
// Listen for responses and distribute them through `activeCounts` // Listen for responses and distribute them through `activeCounts`
this.notifyService.getMessageObservable<CountUserResponse>(RESPONSE_NAME).subscribe(response => { this.notifyService.getMessageObservable<CountUserResponse>(RESPONSE_NAME).subscribe(response => {
if (response.content.data && response.content.token && this.activeCounts[response.content.token]) { if (response.message.data && response.message.token && this.activeCounts[response.message.token]) {
this.activeCounts[response.content.token].next(response.content.data); this.activeCounts[response.message.token].next(response.message.data);
} }
}); });

View File

@ -7,7 +7,7 @@ import { distinctUntilChanged } from 'rxjs/operators';
import { largeDialogSettings } from 'app/shared/utils/dialog-settings'; import { largeDialogSettings } from 'app/shared/utils/dialog-settings';
import { SuperSearchComponent } from 'app/site/common/components/super-search/super-search.component'; import { SuperSearchComponent } from 'app/site/common/components/super-search/super-search.component';
import { DataStoreUpgradeService } from '../core-services/data-store-upgrade.service'; import { DataStoreUpgradeService } from '../core-services/data-store-upgrade.service';
import { OfflineService } from '../core-services/offline.service'; import { OfflineBroadcastService } from '../core-services/offline-broadcast.service';
import { OpenSlidesService } from '../core-services/openslides.service'; import { OpenSlidesService } from '../core-services/openslides.service';
import { OperatorService } from '../core-services/operator.service'; import { OperatorService } from '../core-services/operator.service';
@ -59,7 +59,7 @@ export class OverlayService {
private operator: OperatorService, private operator: OperatorService,
OpenSlides: OpenSlidesService, OpenSlides: OpenSlidesService,
upgradeService: DataStoreUpgradeService, upgradeService: DataStoreUpgradeService,
offlineService: OfflineService offlineBroadcastService: OfflineBroadcastService
) { ) {
// Subscribe to the current user. // Subscribe to the current user.
operator.getViewUserObservable().subscribe(user => { operator.getViewUserObservable().subscribe(user => {
@ -79,10 +79,7 @@ export class OverlayService {
this.checkConnection(); this.checkConnection();
}); });
// Subscribe to check if we are offline // Subscribe to check if we are offline
offlineService offlineBroadcastService.isOfflineObservable.pipe(distinctUntilChanged()).subscribe(offline => {
.isOffline()
.pipe(distinctUntilChanged())
.subscribe(offline => {
this.isOffline = offline; this.isOffline = offline;
this.checkConnection(); this.checkConnection();
}); });

View File

@ -211,15 +211,15 @@ export class C4DialogComponent implements OnInit, OnDestroy {
search: { search: {
recievedSearchRequest: { recievedSearchRequest: {
handle: (notify: NotifyResponse<{ name: string }>) => { handle: (notify: NotifyResponse<{ name: string }>) => {
this.replyChannel = notify.senderChannelName; this.replyChannel = notify.sender_channel_id;
this.partnerName = notify.content.name; this.partnerName = notify.message.name;
return 'waitForResponse'; return 'waitForResponse';
} }
}, },
recievedSearchResponse: { recievedSearchResponse: {
handle: (notify: NotifyResponse<{ name: string }>) => { handle: (notify: NotifyResponse<{ name: string }>) => {
this.replyChannel = notify.senderChannelName; this.replyChannel = notify.sender_channel_id;
this.partnerName = notify.content.name; this.partnerName = notify.message.name;
// who starts? // who starts?
const startPlayer = Math.random() < 0.5 ? Player.thisPlayer : Player.partner; const startPlayer = Math.random() < 0.5 ? Player.thisPlayer : Player.partner;
const startPartner: boolean = startPlayer === Player.partner; const startPartner: boolean = startPlayer === Player.partner;
@ -232,10 +232,10 @@ export class C4DialogComponent implements OnInit, OnDestroy {
waitForResponse: { waitForResponse: {
recievedACK: { recievedACK: {
handle: (notify: NotifyResponse<{}>) => { handle: (notify: NotifyResponse<{}>) => {
if (notify.senderChannelName !== this.replyChannel) { if (notify.sender_channel_id !== this.replyChannel) {
return null; return null;
} }
return notify.content ? 'myTurn' : 'foreignTurn'; return notify.message ? 'myTurn' : 'foreignTurn';
} }
}, },
waitTimeout: { waitTimeout: {
@ -243,7 +243,7 @@ export class C4DialogComponent implements OnInit, OnDestroy {
}, },
recievedRagequit: { recievedRagequit: {
handle: (notify: NotifyResponse<{}>) => { handle: (notify: NotifyResponse<{}>) => {
return notify.senderChannelName === this.replyChannel ? 'search' : null; return notify.sender_channel_id === this.replyChannel ? 'search' : null;
} }
} }
}, },
@ -270,10 +270,10 @@ export class C4DialogComponent implements OnInit, OnDestroy {
foreignTurn: { foreignTurn: {
recievedTurn: { recievedTurn: {
handle: (notify: NotifyResponse<{ col: number }>) => { handle: (notify: NotifyResponse<{ col: number }>) => {
if (notify.senderChannelName !== this.replyChannel) { if (notify.sender_channel_id !== this.replyChannel) {
return null; return null;
} }
const col: number = notify.content.col; const col: number = notify.message.col;
if (!this.colFree(col)) { if (!this.colFree(col)) {
return null; return null;
} }
@ -455,7 +455,7 @@ export class C4DialogComponent implements OnInit, OnDestroy {
*/ */
public enter_waitForResponse(): void { public enter_waitForResponse(): void {
this.caption = 'Wait for response...'; this.caption = 'Wait for response...';
this.notifyService.send('c4_search_response', { name: this.getPlayerName() }); this.notifyService.sendToChannels('c4_search_response', { name: this.getPlayerName() }, this.replyChannel);
if (this.waitTimout) { if (this.waitTimout) {
clearTimeout(<any>this.waitTimout); clearTimeout(<any>this.waitTimout);
} }

View File

@ -28,6 +28,7 @@ import { ViewportService } from 'app/core/ui-services/viewport.service';
import { BaseProjectableViewModel } from 'app/site/base/base-projectable-view-model'; import { BaseProjectableViewModel } from 'app/site/base/base-projectable-view-model';
import { BaseViewModel } from 'app/site/base/base-view-model'; import { BaseViewModel } from 'app/site/base/base-view-model';
import { BaseViewModelWithContentObject } from 'app/site/base/base-view-model-with-content-object'; import { BaseViewModelWithContentObject } from 'app/site/base/base-view-model-with-content-object';
import { isProjectable } from 'app/site/base/projectable';
export interface CssClassDefinition { export interface CssClassDefinition {
[key: string]: boolean; [key: string]: boolean;
@ -458,8 +459,8 @@ export class ListViewTableComponent<V extends BaseViewModel | BaseViewModelWithC
} }
public isElementProjected = (context: PblNgridRowContext<V>) => { public isElementProjected = (context: PblNgridRowContext<V>) => {
const model = context.$implicit as V; const projectableViewModel = this.getProjectable(context.$implicit as V);
if (this.allowProjector && this.projectorService.isProjected(this.getProjectable(model))) { if (projectableViewModel && this.allowProjector && this.projectorService.isProjected(projectableViewModel)) {
return 'projected'; return 'projected';
} }
}; };
@ -578,8 +579,10 @@ export class ListViewTableComponent<V extends BaseViewModel | BaseViewModelWithC
* @param viewModel The model of the table * @param viewModel The model of the table
* @returns a view model that can be projected * @returns a view model that can be projected
*/ */
public getProjectable(viewModel: V): BaseProjectableViewModel { public getProjectable(viewModel: V): BaseProjectableViewModel | null {
return (viewModel as BaseViewModelWithContentObject)?.contentObject ?? viewModel; const actualViewModel: BaseProjectableViewModel =
(viewModel as BaseViewModelWithContentObject)?.contentObject ?? viewModel;
return isProjectable(actualViewModel) ? actualViewModel : null;
} }
/** /**

View File

@ -5,7 +5,7 @@ import { TranslateService } from '@ngx-translate/core';
import { Subject, Subscription } from 'rxjs'; import { Subject, Subscription } from 'rxjs';
import { BaseComponent } from 'app/base.component'; import { BaseComponent } from 'app/base.component';
import { OfflineService } from 'app/core/core-services/offline.service'; import { OfflineBroadcastService } from 'app/core/core-services/offline-broadcast.service';
import { ProjectorDataService, SlideData } from 'app/core/core-services/projector-data.service'; import { ProjectorDataService, SlideData } from 'app/core/core-services/projector-data.service';
import { ProjectorRepositoryService } from 'app/core/repositories/projector/projector-repository.service'; import { ProjectorRepositoryService } from 'app/core/repositories/projector/projector-repository.service';
import { ConfigService } from 'app/core/ui-services/config.service'; import { ConfigService } from 'app/core/ui-services/config.service';
@ -166,7 +166,7 @@ export class ProjectorComponent extends BaseComponent implements OnDestroy {
private projectorDataService: ProjectorDataService, private projectorDataService: ProjectorDataService,
private projectorRepository: ProjectorRepositoryService, private projectorRepository: ProjectorRepositoryService,
private configService: ConfigService, private configService: ConfigService,
private offlineService: OfflineService, private offlineBroadcastService: OfflineBroadcastService,
private elementRef: ElementRef private elementRef: ElementRef
) { ) {
super(titleService, translate); super(titleService, translate);
@ -207,7 +207,9 @@ export class ProjectorComponent extends BaseComponent implements OnDestroy {
} }
}); });
this.offlineSubscription = this.offlineService.isOffline().subscribe(isOffline => (this.isOffline = isOffline)); this.offlineSubscription = this.offlineBroadcastService.isOfflineObservable.subscribe(
isOffline => (this.isOffline = isOffline)
);
} }
/** /**

View File

@ -13,6 +13,7 @@
</os-head-bar> </os-head-bar>
<mat-card class="spacer-bottom-60" [ngClass]="isEditing ? 'os-form-card' : 'os-card'"> <mat-card class="spacer-bottom-60" [ngClass]="isEditing ? 'os-form-card' : 'os-card'">
<mat-card [ngClass]="isEditing ? 'os-form-card' : 'os-card'">
<ng-container *ngIf="!isEditing"> <ng-container *ngIf="!isEditing">
<div class="app-content"> <div class="app-content">
<h1>{{ startContent.general_event_welcome_title | translate }}</h1> <h1>{{ startContent.general_event_welcome_title | translate }}</h1>

View File

@ -1639,7 +1639,7 @@ export class MotionDetailComponent extends BaseViewComponentDirective implements
*/ */
private listenToEditNotification(): Subscription { private listenToEditNotification(): Subscription {
return this.notifyService.getMessageObservable(this.NOTIFICATION_EDIT_MOTION).subscribe(message => { return this.notifyService.getMessageObservable(this.NOTIFICATION_EDIT_MOTION).subscribe(message => {
const content = <MotionEditNotification>message.content; const content = <MotionEditNotification>message.message;
if (this.operator.viewUser.id !== content.senderId && content.motionId === this.motion.id) { if (this.operator.viewUser.id !== content.senderId && content.motionId === this.motion.id) {
let warning = ''; let warning = '';
@ -1656,7 +1656,7 @@ export class MotionDetailComponent extends BaseViewComponentDirective implements
if (content.type === MotionEditNotificationType.TYPE_BEGIN_EDITING_MOTION) { if (content.type === MotionEditNotificationType.TYPE_BEGIN_EDITING_MOTION) {
this.sendEditNotification( this.sendEditNotification(
MotionEditNotificationType.TYPE_ALSO_EDITING_MOTION, MotionEditNotificationType.TYPE_ALSO_EDITING_MOTION,
message.senderUserId message.sender_user_id
); );
} }
break; break;

View File

@ -58,8 +58,6 @@ export abstract class BasePollDialogComponent<
votes: this.getVoteData(), votes: this.getVoteData(),
publish_immediately: this.publishImmediately publish_immediately: this.publishImmediately
}; };
console.log('answer: ', answer);
this.dialogRef.close(answer); this.dialogRef.close(answer);
} }

View File

@ -10,7 +10,7 @@ import { TranslateService } from '@ngx-translate/core';
import { filter } from 'rxjs/operators'; import { filter } from 'rxjs/operators';
import { navItemAnim } from '../shared/animations'; import { navItemAnim } from '../shared/animations';
import { OfflineService } from 'app/core/core-services/offline.service'; import { OfflineBroadcastService } from 'app/core/core-services/offline-broadcast.service';
import { OverlayService } from 'app/core/ui-services/overlay.service'; import { OverlayService } from 'app/core/ui-services/overlay.service';
import { UpdateService } from 'app/core/ui-services/update.service'; import { UpdateService } from 'app/core/ui-services/update.service';
import { BaseComponent } from '../base.component'; import { BaseComponent } from '../base.component';
@ -84,7 +84,7 @@ export class SiteComponent extends BaseComponent implements OnInit {
public constructor( public constructor(
title: Title, title: Title,
protected translate: TranslateService, protected translate: TranslateService,
offlineService: OfflineService, offlineBroadcastService: OfflineBroadcastService,
private updateService: UpdateService, private updateService: UpdateService,
private router: Router, private router: Router,
public operator: OperatorService, public operator: OperatorService,
@ -99,7 +99,7 @@ export class SiteComponent extends BaseComponent implements OnInit {
super(title, translate); super(title, translate);
overlayService.showSpinner(translate.instant('Loading data. Please wait ...')); overlayService.showSpinner(translate.instant('Loading data. Please wait ...'));
offlineService.isOffline().subscribe(offline => { offlineBroadcastService.isOfflineObservable.subscribe(offline => {
this.isOffline = offline; this.isOffline = offline;
}); });

3
dc-dev.sh Executable file
View File

@ -0,0 +1,3 @@
#!/bin/bash
cd "$(dirname $0)"
docker-compose -f docker/docker-compose.dev.yml $@

View File

@ -17,10 +17,14 @@ DEFAULT_DOCKER_REGISTRY=
# Docker Images # Docker Images
# ------------- # -------------
DOCKER_OPENSLIDES_HAPROXY_NAME=
DOCKER_OPENSLIDES_HAPROXY_TAG=
DOCKER_OPENSLIDES_BACKEND_NAME= DOCKER_OPENSLIDES_BACKEND_NAME=
DOCKER_OPENSLIDES_BACKEND_TAG= DOCKER_OPENSLIDES_BACKEND_TAG=
DOCKER_OPENSLIDES_FRONTEND_NAME= DOCKER_OPENSLIDES_FRONTEND_NAME=
DOCKER_OPENSLIDES_FRONTEND_TAG= DOCKER_OPENSLIDES_FRONTEND_TAG=
DOCKER_OPENSLIDES_AUTOUPDATE_NAME=
DOCKER_OPENSLIDES_AUTOUPDATE_TAG=
# Database # Database
# -------- # --------
@ -37,6 +41,7 @@ PGBOUNCER_PLACEMENT_CONSTR=
# ------------------- # -------------------
OPENSLIDES_BACKEND_SERVICE_REPLICAS= OPENSLIDES_BACKEND_SERVICE_REPLICAS=
OPENSLIDES_FRONTEND_SERVICE_REPLICAS= OPENSLIDES_FRONTEND_SERVICE_REPLICAS=
OPENSLIDES_AUTOUPDATE_SERVICE_REPLICAS=
REDIS_RO_SERVICE_REPLICAS= REDIS_RO_SERVICE_REPLICAS=
MEDIA_SERVICE_REPLICAS= MEDIA_SERVICE_REPLICAS=

View File

@ -6,7 +6,9 @@ declare -A TARGETS
TARGETS=( TARGETS=(
[client]="$(dirname "${BASH_SOURCE[0]}")/../client/docker/" [client]="$(dirname "${BASH_SOURCE[0]}")/../client/docker/"
[server]="$(dirname "${BASH_SOURCE[0]}")/../server/docker/" [server]="$(dirname "${BASH_SOURCE[0]}")/../server/docker/"
[media-service]="https://github.com/OpenSlides/openslides-media-service.git" [haproxy]="$(dirname "${BASH_SOURCE[0]}")/../haproxy/"
[autoupdate]="$(dirname "${BASH_SOURCE[0]}")/../autoupdate/"
[media]="https://github.com/OpenSlides/openslides-media-service.git"
[pgbouncer]="https://github.com/OpenSlides/openslides-docker-compose.git#:pgbouncer" [pgbouncer]="https://github.com/OpenSlides/openslides-docker-compose.git#:pgbouncer"
[postfix]="https://github.com/OpenSlides/openslides-docker-compose.git#:postfix" [postfix]="https://github.com/OpenSlides/openslides-docker-compose.git#:postfix"
[repmgr]="https://github.com/OpenSlides/openslides-docker-compose.git#:repmgr" [repmgr]="https://github.com/OpenSlides/openslides-docker-compose.git#:repmgr"
@ -17,7 +19,7 @@ DOCKER_TAG="latest"
CONFIG="/etc/osinstancectl" CONFIG="/etc/osinstancectl"
OPTIONS=() OPTIONS=()
BUILT_IMAGES=() BUILT_IMAGES=()
DEFAULT_TARGETS=(server client) DEFAULT_TARGETS=(server client haproxy autoupdate)
usage() { usage() {
cat << EOF cat << EOF

View File

@ -0,0 +1,47 @@
version: "3"
services:
client:
image: os3-client-dev
build:
context: ../client
dockerfile: docker/Dockerfile.dev
volumes:
- ../client:/app
server:
image: os3-server-dev
user: $UID:$GID
build:
context: ../server
dockerfile: docker/Dockerfile.dev
volumes:
- ../server:/app
depends_on:
- redis
autoupdate:
image: os3-autoupdate-dev
environment:
- MESSAGE_BUS_HOST=redis
- MESSAGE_BUS_PORT=6379
- WORKER_HOST=server
- WORKER_PORT=8000
depends_on:
- server
- redis
volumes:
- ../autoupdate/cmd:/root/cmd
- ../autoupdate/internal:/root/internal
redis:
image: redis:latest
haproxy:
image: os3-haproxy-dev
volumes:
- ../haproxy/src:/usr/local/etc/haproxy
depends_on:
- client
- server
- autoupdate
ports:
- "8000:8000"

View File

@ -13,12 +13,22 @@ define(`read_env', `esyscmd(`printf "\`%s'" "$$1"')')
dnl return env variable if set; otherwise, return given alternative value dnl return env variable if set; otherwise, return given alternative value
define(`ifenvelse', `ifelse(read_env(`$1'),, `$2', read_env(`$1'))') define(`ifenvelse', `ifelse(read_env(`$1'),, `$2', read_env(`$1'))')
define(`HAPROXY_IMAGE',
ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_NAME', openslides-haproxy):dnl
ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_TAG', latest))
define(`BACKEND_IMAGE', define(`BACKEND_IMAGE',
ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides/openslides-server):dnl ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides-server):dnl
ifenvelse(`DOCKER_OPENSLIDES_BACKEND_TAG', latest)) ifenvelse(`DOCKER_OPENSLIDES_BACKEND_TAG', latest))
define(`FRONTEND_IMAGE', define(`FRONTEND_IMAGE',
ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides/openslides-client):dnl ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides-client):dnl
ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_TAG', latest)) ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_TAG', latest))
define(`AUTOUPDATE_IMAGE',
ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_NAME', openslides-autoupdate):dnl
ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_TAG', latest))
define(`PRIMARY_DB', `ifenvelse(`PGNODE_REPMGR_PRIMARY', pgnode1)') define(`PRIMARY_DB', `ifenvelse(`PGNODE_REPMGR_PRIMARY', pgnode1)')
@ -41,11 +51,9 @@ x-osserver:
&default-osserver &default-osserver
image: BACKEND_IMAGE image: BACKEND_IMAGE
networks: networks:
- front
- back - back
restart: always restart: always
x-osserver-env: &default-osserver-env x-osserver-env: &default-osserver-env
AMOUNT_REPLICAS: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1)
AUTOUPDATE_DELAY: ifenvelse(`AUTOUPDATE_DELAY', 1) AUTOUPDATE_DELAY: ifenvelse(`AUTOUPDATE_DELAY', 1)
DEMO_USERS: "ifenvelse(`DEMO_USERS',)" DEMO_USERS: "ifenvelse(`DEMO_USERS',)"
CONNECTION_POOL_LIMIT: ifenvelse(`CONNECTION_POOL_LIMIT', 100) CONNECTION_POOL_LIMIT: ifenvelse(`CONNECTION_POOL_LIMIT', 100)
@ -54,7 +62,6 @@ x-osserver-env: &default-osserver-env
DATABASE_PORT: ifenvelse(`DATABASE_PORT', 5432) DATABASE_PORT: ifenvelse(`DATABASE_PORT', 5432)
DATABASE_USER: "ifenvelse(`DATABASE_USER', openslides)" DATABASE_USER: "ifenvelse(`DATABASE_USER', openslides)"
DEFAULT_FROM_EMAIL: "ifenvelse(`DEFAULT_FROM_EMAIL', noreply@example.com)" DEFAULT_FROM_EMAIL: "ifenvelse(`DEFAULT_FROM_EMAIL', noreply@example.com)"
DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
EMAIL_HOST: "ifenvelse(`EMAIL_HOST', postfix)" EMAIL_HOST: "ifenvelse(`EMAIL_HOST', postfix)"
EMAIL_HOST_PASSWORD: "ifenvelse(`EMAIL_HOST_PASSWORD',)" EMAIL_HOST_PASSWORD: "ifenvelse(`EMAIL_HOST_PASSWORD',)"
EMAIL_HOST_USER: "ifenvelse(`EMAIL_HOST_USER',)" EMAIL_HOST_USER: "ifenvelse(`EMAIL_HOST_USER',)"
@ -69,13 +76,11 @@ x-osserver-env: &default-osserver-env
JITSI_ROOM_PASSWORD: "ifenvelse(`JITSI_ROOM_PASSWORD',)" JITSI_ROOM_PASSWORD: "ifenvelse(`JITSI_ROOM_PASSWORD',)"
JITSI_ROOM_NAME: "ifenvelse(`JITSI_ROOM_NAME',)" JITSI_ROOM_NAME: "ifenvelse(`JITSI_ROOM_NAME',)"
OPENSLIDES_LOG_LEVEL: "ifenvelse(`OPENSLIDES_LOG_LEVEL', INFO)" OPENSLIDES_LOG_LEVEL: "ifenvelse(`OPENSLIDES_LOG_LEVEL', INFO)"
REDIS_CHANNLES_HOST: "ifenvelse(`REDIS_CHANNLES_HOST', redis-channels)" DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
REDIS_CHANNLES_PORT: ifenvelse(`REDIS_CHANNLES_PORT', 6379)
REDIS_HOST: "ifenvelse(`REDIS_HOST', redis)" REDIS_HOST: "ifenvelse(`REDIS_HOST', redis)"
REDIS_PORT: ifenvelse(`REDIS_PORT', 6379) REDIS_PORT: ifenvelse(`REDIS_PORT', 6379)
REDIS_SLAVE_HOST: "ifenvelse(`REDIS_SLAVE_HOST', redis-slave)" REDIS_SLAVE_HOST: "ifenvelse(`REDIS_SLAVE_HOST', redis-slave)"
REDIS_SLAVE_PORT: ifenvelse(`REDIS_SLAVE_PORT', 6379) REDIS_SLAVE_PORT: ifenvelse(`REDIS_SLAVE_PORT', 6379)
REDIS_SLAVE_WAIT_TIMEOUT: ifenvelse(`REDIS_SLAVE_WAIT_TIMEOUT', 10000)
RESET_PASSWORD_VERBOSE_ERRORS: "ifenvelse(`RESET_PASSWORD_VERBOSE_ERRORS', False)" RESET_PASSWORD_VERBOSE_ERRORS: "ifenvelse(`RESET_PASSWORD_VERBOSE_ERRORS', False)"
x-pgnode: &default-pgnode x-pgnode: &default-pgnode
image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-repmgr:latest image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-repmgr:latest
@ -90,15 +95,27 @@ x-pgnode-env: &default-pgnode-env
REPMGR_WAL_ARCHIVE: "ifenvelse(`PGNODE_WAL_ARCHIVING', on)" REPMGR_WAL_ARCHIVE: "ifenvelse(`PGNODE_WAL_ARCHIVING', on)"
services: services:
haproxy:
image: HAPROXY_IMAGE
depends_on:
- server
- client
- autoupdate
- media
networks:
- front
- back
ports:
- "127.0.0.1:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):8000"
server: server:
<< : *default-osserver << : *default-osserver
# Below is the default command. You can uncomment it to override the # Below is the default command. You can uncomment it to override the
# number of workers, for example: # number of workers, for example:
# command: "gunicorn -w 8 --preload -b 0.0.0.0:8000 # command: "gunicorn -w 8 --preload -b 0.0.0.0:8000 openslides.wsgi"
# -k uvicorn.workers.UvicornWorker openslides.asgi:application"
# #
# Uncomment the following line to use daphne instead of gunicorn: # Uncomment the following line to use daphne instead of gunicorn:
# command: "daphne -b 0.0.0.0 -p 8000 openslides.asgi:application" # command: "daphne -b 0.0.0.0 -p 8000 openslides.wsgi"
depends_on: depends_on:
- server-setup - server-setup
environment: environment:
@ -127,17 +144,24 @@ services:
- pgbouncer - pgbouncer
- redis - redis
- redis-slave - redis-slave
- redis-channels
client: client:
image: FRONTEND_IMAGE image: FRONTEND_IMAGE
restart: always restart: always
depends_on:
- server
networks: networks:
- front - back
ports:
- "127.0.0.1:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):80" autoupdate:
image: AUTOUPDATE_IMAGE
restart: always
depends_on:
- redis
- server
environment:
MESSAGE_BUS_HOST: redis
WORKER_HOST: server
networks:
- back
pgnode1: pgnode1:
<< : *default-pgnode << : *default-pgnode
@ -189,9 +213,7 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
image: redis:alpine image: redis:alpine
restart: always restart: always
networks: networks:
back: - back
aliases:
- rediscache
redis-slave: redis-slave:
image: redis:alpine image: redis:alpine
restart: always restart: always
@ -199,18 +221,11 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
depends_on: depends_on:
- redis - redis
networks: networks:
back: - back
aliases:
- rediscache-slave
ifelse(read_env(`REDIS_RO_SERVICE_REPLICAS'),,,deploy: ifelse(read_env(`REDIS_RO_SERVICE_REPLICAS'),,,deploy:
replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1)) replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1))
redis-channels:
image: redis:alpine
restart: always
networks:
back:
media: media:
image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media-service:latest image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media:latest
environment: environment:
- CHECK_REQUEST_URL=server:8000/check-media/ - CHECK_REQUEST_URL=server:8000/check-media/
- CACHE_SIZE=ifenvelse(`CACHE_SIZE', 10) - CACHE_SIZE=ifenvelse(`CACHE_SIZE', 10)
@ -218,8 +233,7 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
- CACHE_DATA_MAX_SIZE_KB=ifenvelse(`CACHE_DATA_MAX_SIZE_KB', 10240) - CACHE_DATA_MAX_SIZE_KB=ifenvelse(`CACHE_DATA_MAX_SIZE_KB', 10240)
restart: always restart: always
networks: networks:
front: - back
back:
# Override command to run more workers per task # Override command to run more workers per task
# command: ["gunicorn", "-w", "4", "--preload", "-b", # command: ["gunicorn", "-w", "4", "--preload", "-b",
# "0.0.0.0:8000", "src.mediaserver:app"] # "0.0.0.0:8000", "src.mediaserver:app"]

View File

@ -13,12 +13,22 @@ define(`read_env', `esyscmd(`printf "\`%s'" "$$1"')')
dnl return env variable if set; otherwise, return given alternative value dnl return env variable if set; otherwise, return given alternative value
define(`ifenvelse', `ifelse(read_env(`$1'),, `$2', read_env(`$1'))') define(`ifenvelse', `ifelse(read_env(`$1'),, `$2', read_env(`$1'))')
define(`HAPROXY_IMAGE',
ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_NAME', openslides-haproxy):dnl
ifenvelse(`DOCKER_OPENSLIDES_HAPROXY_TAG', latest))
define(`BACKEND_IMAGE', define(`BACKEND_IMAGE',
ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides/openslides-server):dnl ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_BACKEND_NAME', openslides-server):dnl
ifenvelse(`DOCKER_OPENSLIDES_BACKEND_TAG', latest)) ifenvelse(`DOCKER_OPENSLIDES_BACKEND_TAG', latest))
define(`FRONTEND_IMAGE', define(`FRONTEND_IMAGE',
ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides/openslides-client):dnl ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_NAME', openslides-client):dnl
ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_TAG', latest)) ifenvelse(`DOCKER_OPENSLIDES_FRONTEND_TAG', latest))
define(`AUTOUPDATE_IMAGE',
ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/dnl
ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_NAME', openslides-autoupdate):dnl
ifenvelse(`DOCKER_OPENSLIDES_AUTOUPDATE_TAG', latest))
define(`PRIMARY_DB', `ifenvelse(`PGNODE_REPMGR_PRIMARY', pgnode1)') define(`PRIMARY_DB', `ifenvelse(`PGNODE_REPMGR_PRIMARY', pgnode1)')
@ -41,10 +51,8 @@ x-osserver:
&default-osserver &default-osserver
image: BACKEND_IMAGE image: BACKEND_IMAGE
networks: networks:
- front
- back - back
x-osserver-env: &default-osserver-env x-osserver-env: &default-osserver-env
AMOUNT_REPLICAS: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 3)
AUTOUPDATE_DELAY: ifenvelse(`AUTOUPDATE_DELAY', 1) AUTOUPDATE_DELAY: ifenvelse(`AUTOUPDATE_DELAY', 1)
DEMO_USERS: "ifenvelse(`DEMO_USERS',)" DEMO_USERS: "ifenvelse(`DEMO_USERS',)"
CONNECTION_POOL_LIMIT: ifenvelse(`CONNECTION_POOL_LIMIT', 100) CONNECTION_POOL_LIMIT: ifenvelse(`CONNECTION_POOL_LIMIT', 100)
@ -53,7 +61,6 @@ x-osserver-env: &default-osserver-env
DATABASE_PORT: ifenvelse(`DATABASE_PORT', 5432) DATABASE_PORT: ifenvelse(`DATABASE_PORT', 5432)
DATABASE_USER: "ifenvelse(`DATABASE_USER', openslides)" DATABASE_USER: "ifenvelse(`DATABASE_USER', openslides)"
DEFAULT_FROM_EMAIL: "ifenvelse(`DEFAULT_FROM_EMAIL', noreply@example.com)" DEFAULT_FROM_EMAIL: "ifenvelse(`DEFAULT_FROM_EMAIL', noreply@example.com)"
DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
EMAIL_HOST: "ifenvelse(`EMAIL_HOST', postfix)" EMAIL_HOST: "ifenvelse(`EMAIL_HOST', postfix)"
EMAIL_HOST_PASSWORD: "ifenvelse(`EMAIL_HOST_PASSWORD',)" EMAIL_HOST_PASSWORD: "ifenvelse(`EMAIL_HOST_PASSWORD',)"
EMAIL_HOST_USER: "ifenvelse(`EMAIL_HOST_USER',)" EMAIL_HOST_USER: "ifenvelse(`EMAIL_HOST_USER',)"
@ -68,13 +75,11 @@ x-osserver-env: &default-osserver-env
JITSI_ROOM_PASSWORD: "ifenvelse(`JITSI_ROOM_PASSWORD',)" JITSI_ROOM_PASSWORD: "ifenvelse(`JITSI_ROOM_PASSWORD',)"
JITSI_ROOM_NAME: "ifenvelse(`JITSI_ROOM_NAME',)" JITSI_ROOM_NAME: "ifenvelse(`JITSI_ROOM_NAME',)"
OPENSLIDES_LOG_LEVEL: "ifenvelse(`OPENSLIDES_LOG_LEVEL', INFO)" OPENSLIDES_LOG_LEVEL: "ifenvelse(`OPENSLIDES_LOG_LEVEL', INFO)"
REDIS_CHANNLES_HOST: "ifenvelse(`REDIS_CHANNLES_HOST', redis-channels)" DJANGO_LOG_LEVEL: "ifenvelse(`DJANGO_LOG_LEVEL', INFO)"
REDIS_CHANNLES_PORT: ifenvelse(`REDIS_CHANNLES_PORT', 6379)
REDIS_HOST: "ifenvelse(`REDIS_HOST', redis)" REDIS_HOST: "ifenvelse(`REDIS_HOST', redis)"
REDIS_PORT: ifenvelse(`REDIS_PORT', 6379) REDIS_PORT: ifenvelse(`REDIS_PORT', 6379)
REDIS_SLAVE_HOST: "ifenvelse(`REDIS_SLAVE_HOST', redis-slave)" REDIS_SLAVE_HOST: "ifenvelse(`REDIS_SLAVE_HOST', redis-slave)"
REDIS_SLAVE_PORT: ifenvelse(`REDIS_SLAVE_PORT', 6379) REDIS_SLAVE_PORT: ifenvelse(`REDIS_SLAVE_PORT', 6379)
REDIS_SLAVE_WAIT_TIMEOUT: ifenvelse(`REDIS_SLAVE_WAIT_TIMEOUT', 10000)
RESET_PASSWORD_VERBOSE_ERRORS: "ifenvelse(`RESET_PASSWORD_VERBOSE_ERRORS', False)" RESET_PASSWORD_VERBOSE_ERRORS: "ifenvelse(`RESET_PASSWORD_VERBOSE_ERRORS', False)"
x-pgnode: &default-pgnode x-pgnode: &default-pgnode
image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-repmgr:latest image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-repmgr:latest
@ -90,6 +95,18 @@ x-pgnode-env: &default-pgnode-env
REPMGR_WAL_ARCHIVE: "ifenvelse(`PGNODE_WAL_ARCHIVING', on)" REPMGR_WAL_ARCHIVE: "ifenvelse(`PGNODE_WAL_ARCHIVING', on)"
services: services:
haproxy:
image: HAPROXY_IMAGE
networks:
- front
- back
ports:
- "0.0.0.0:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):8000"
deploy:
restart_policy:
condition: on-failure
delay: 5s
server: server:
<< : *default-osserver << : *default-osserver
# Below is the default command. You can uncomment it to override the # Below is the default command. You can uncomment it to override the
@ -128,15 +145,26 @@ services:
client: client:
image: FRONTEND_IMAGE image: FRONTEND_IMAGE
networks: networks:
- front - back
ports:
- "0.0.0.0:ifenvelse(`EXTERNAL_HTTP_PORT', 8000):80"
deploy: deploy:
replicas: ifenvelse(`OPENSLIDES_FRONTEND_SERVICE_REPLICAS', 1) replicas: ifenvelse(`OPENSLIDES_FRONTEND_SERVICE_REPLICAS', 1)
restart_policy: restart_policy:
condition: on-failure condition: on-failure
delay: 5s delay: 5s
autoupdate:
image: AUTOUPDATE_IMAGE
environment:
MESSAGE_BUS_HOST: redis
WORKER_HOST: server
networks:
- back
deploy:
replicas: ifenvelse(`OPENSLIDES_AUTOUPDATE_SERVICE_REPLICAS', 1)
restart_policy:
condition: on-failure
delay: 5s
pgnode1: pgnode1:
<< : *default-pgnode << : *default-pgnode
environment: environment:
@ -206,9 +234,7 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
redis: redis:
image: redis:alpine image: redis:alpine
networks: networks:
back: - back
aliases:
- rediscache
deploy: deploy:
replicas: 1 replicas: 1
restart_policy: restart_policy:
@ -218,38 +244,26 @@ ifelse(read_env(`PGNODE_3_ENABLED'), 1, `'
image: redis:alpine image: redis:alpine
command: ["redis-server", "--save", "", "--slaveof", "redis", "6379"] command: ["redis-server", "--save", "", "--slaveof", "redis", "6379"]
networks: networks:
back: - back
aliases:
- rediscache-slave
deploy: deploy:
replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 3) replicas: ifenvelse(`REDIS_RO_SERVICE_REPLICAS', 1)
restart_policy:
condition: on-failure
delay: 5s
redis-channels:
image: redis:alpine
networks:
back:
deploy:
replicas: 1
restart_policy: restart_policy:
condition: on-failure condition: on-failure
delay: 5s delay: 5s
media: media:
image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media-service:latest image: ifenvelse(`DEFAULT_DOCKER_REGISTRY', openslides)/openslides-media:latest
environment: environment:
- CHECK_REQUEST_URL=server:8000/check-media/ - CHECK_REQUEST_URL=server:8000/check-media/
- CACHE_SIZE=ifenvelse(`CACHE_SIZE', 10) - CACHE_SIZE=ifenvelse(`CACHE_SIZE', 10)
- CACHE_DATA_MIN_SIZE_KB=ifenvelse(`CACHE_DATA_MIN_SIZE_KB', 0) - CACHE_DATA_MIN_SIZE_KB=ifenvelse(`CACHE_DATA_MIN_SIZE_KB', 0)
- CACHE_DATA_MAX_SIZE_KB=ifenvelse(`CACHE_DATA_MAX_SIZE_KB', 10240) - CACHE_DATA_MAX_SIZE_KB=ifenvelse(`CACHE_DATA_MAX_SIZE_KB', 10240)
deploy: deploy:
replicas: ifenvelse(`MEDIA_SERVICE_REPLICAS', 8) replicas: ifenvelse(`MEDIA_SERVICE_REPLICAS', 2)
restart_policy: restart_policy:
condition: on-failure condition: on-failure
delay: 10s delay: 10s
networks: networks:
front: - back
back:
# Override command to run more workers per task # Override command to run more workers per task
# command: ["gunicorn", "-w", "4", "--preload", "-b", # command: ["gunicorn", "-w", "4", "--preload", "-b",
# "0.0.0.0:8000", "src.mediaserver:app"] # "0.0.0.0:8000", "src.mediaserver:app"]

5
haproxy/Dockerfile Normal file
View File

@ -0,0 +1,5 @@
FROM haproxy:2.0-alpine
COPY src/haproxy.common.cfg /usr/local/etc/haproxy/haproxy.common.cfg
COPY src/haproxy.prod.cfg /usr/local/etc/haproxy/haproxy.prod.cfg
COPY src/combined.pem /usr/local/etc/haproxy/combined.pem
CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.common.cfg", "-f", "/usr/local/etc/haproxy/haproxy.prod.cfg"]

5
haproxy/Dockerfile.dev Normal file
View File

@ -0,0 +1,5 @@
FROM haproxy:2.0-alpine
COPY src/haproxy.common.cfg /usr/local/etc/haproxy/haproxy.common.cfg
COPY src/haproxy.dev.cfg /usr/local/etc/haproxy/haproxy.dev.cfg
COPY src/combined.pem /usr/local/etc/haproxy/combined.pem
CMD ["haproxy", "-f", "/usr/local/etc/haproxy/haproxy.common.cfg", "-f", "/usr/local/etc/haproxy/haproxy.dev.cfg"]

3
haproxy/Makefile Normal file
View File

@ -0,0 +1,3 @@
build-dev:
./prepare-cert.sh
docker build -t os3-haproxy-dev -f Dockerfile.dev .

6
haproxy/build.sh Executable file
View File

@ -0,0 +1,6 @@
#!/bin/bash
cd "$(dirname "${BASH_SOURCE[0]}")"
./prepare-cert.sh
docker build --tag "${img:-openslides/openslides-${service_name}:latest}" \
--pull "${OPTIONS[@]}" .

27
haproxy/prepare-cert.sh Executable file
View File

@ -0,0 +1,27 @@
#!/bin/bash
set -e
cd "$(dirname "$0")"
# check, if we already generated a cert
combined="src/combined.pem"
if [[ ! -f $combined ]]; then
echo "Creating certificates..."
cd src
if type 2>&1 >/dev/null openssl ; then
echo "Using openssl to generate a certificate."
echo "You will need to accept an security exception for the"
echo "generated certificate in your browser manually."
openssl req -x509 -newkey rsa:4096 -nodes -days 3650 \
-subj "/C=DE/O=Selfsigned Test/CN=localhost" \
-keyout localhost-key.pem -out localhost.pem
else
echo >&2 "FATAL: No valid certificate generation tool found!"
exit -1
fi
cat localhost.pem localhost-key.pem > combined.pem
echo "done"
else
echo "Certificate exists."
fi

View File

@ -0,0 +1,31 @@
global
log stdout format raw local0 debug
defaults
option http-use-htx
option dontlognull
timeout connect 3s
timeout client 10s
timeout client-fin 10s
timeout server 10s
timeout server-fin 10s
timeout check 2s
timeout tunnel 10s
timeout queue 2s
log global
option httplog
resolvers docker_resolver
nameserver dns 127.0.0.11:53
backend backend_server
mode http
# Do not pass the auth-header from /stats to OS. It confuses the server...
http-request del-header authorization
timeout server 4m
server server server:8000 resolvers docker_resolver check alpn http/1.1
backend backend_autoupdate
mode http
timeout server 1h
server autoupdate autoupdate:8002 resolvers docker_resolver check ssl verify none alpn h2

View File

@ -0,0 +1,20 @@
frontend https
mode http
bind *:8000 ssl crt /usr/local/etc/haproxy/combined.pem alpn h2,http/1.1
default_backend backend_client
acl autoupdate path_beg -i /system
use_backend backend_autoupdate if autoupdate
acl server path_beg -i /apps /media/ /rest /server-version.txt
use_backend backend_server if server
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:admin
backend backend_client
mode http
timeout tunnel 1h
server client client:4200 resolvers docker_resolver no-check

View File

@ -0,0 +1,26 @@
frontend https
mode http
bind *:8000 ssl crt /usr/local/etc/haproxy/combined.pem alpn h2,http/1.1
default_backend backend_client
acl autoupdate path_beg -i /system
use_backend backend_autoupdate if autoupdate
acl server path_beg -i /apps /rest /server-version.txt
use_backend backend_server if server
acl media path_beg -i /media/
use_backend backend_media if media
stats enable
stats uri /stats
stats refresh 10s
stats auth admin:admin
backend backend_client
mode http
server client client:80 resolvers docker_resolver check
backend backend_media
mode http
server media media:8000 resolvers docker_resolver check

View File

@ -3,3 +3,6 @@
**/.venv **/.venv
tests/ tests/
personal_data/ personal_data/
**/.mypy_cache
**/.pytest_cache
**/tests/file

View File

@ -70,5 +70,4 @@ COPY manage.py /app/
COPY openslides /app/openslides COPY openslides /app/openslides
COPY docker/server-version.txt /app/openslides/core/static/server-version.txt COPY docker/server-version.txt /app/openslides/core/static/server-version.txt
ENTRYPOINT ["/usr/local/sbin/entrypoint"] ENTRYPOINT ["/usr/local/sbin/entrypoint"]
CMD ["gunicorn", "-w", "8", "--preload", "-b", "0.0.0.0:8000", "-k", \ CMD ["gunicorn", "-w", "8", "--preload", "-t", "240", "-b", "0.0.0.0:8000", "openslides.wsgi"]
"uvicorn.workers.UvicornWorker", "openslides.asgi:application"]

View File

@ -0,0 +1,30 @@
FROM python:3.7-slim AS base
# Variables relevant for CMD
ENV DJANGO_SETTINGS_MODULE settings
ENV PYTHONPATH personal_data/var/
WORKDIR /app
RUN apt-get -y update && apt-get install --no-install-recommends -y \
postgresql-client \
wait-for-it \
gcc \
git \
libxml2-dev \
libxmlsec1-dev \
libxmlsec1-openssl \
pkg-config
RUN rm -rf /var/lib/apt/lists/*
COPY requirements /app/requirements
COPY requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt -r requirements/saml.txt && \
rm -rf /root/.cache/pip
EXPOSE 8000
COPY docker/entrypoint-dev /usr/local/sbin/
COPY . .
ENTRYPOINT ["/usr/local/sbin/entrypoint-dev"]
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

View File

@ -37,7 +37,6 @@ done
# Wait for redis # Wait for redis
wait-for-it redis:6379 wait-for-it redis:6379
wait-for-it redis-slave:6379 wait-for-it redis-slave:6379
wait-for-it redis-channels:6379
echo 'running migrations' echo 'running migrations'
python manage.py migrate python manage.py migrate

14
server/docker/entrypoint-dev Executable file
View File

@ -0,0 +1,14 @@
#!/bin/bash
set -e
wait-for-it -t 0 redis:6379
if [[ ! -f "/app/personal_data/var/settings.py" ]]; then
echo "Create settings"
python manage.py createsettings
fi
python -u manage.py migrate
exec "$@"

View File

@ -54,7 +54,7 @@ DEBUG = False
RESET_PASSWORD_VERBOSE_ERRORS = get_env("RESET_PASSWORD_VERBOSE_ERRORS", True, bool) RESET_PASSWORD_VERBOSE_ERRORS = get_env("RESET_PASSWORD_VERBOSE_ERRORS", True, bool)
# OpenSlides specific settings # OpenSlides specific settings
AUTOUPDATE_DELAY = get_env("AUTOUPDATE_DELAY", 1, int) AUTOUPDATE_DELAY = get_env("AUTOUPDATE_DELAY", 1, float)
DEMO_USERS = get_env("DEMO_USERS", default=None) DEMO_USERS = get_env("DEMO_USERS", default=None)
DEMO_USERS = json.loads(DEMO_USERS) if DEMO_USERS else None DEMO_USERS = json.loads(DEMO_USERS) if DEMO_USERS else None
@ -99,25 +99,10 @@ REDIS_HOST = get_env("REDIS_HOST", "redis")
REDIS_PORT = get_env("REDIS_PORT", 6379, int) REDIS_PORT = get_env("REDIS_PORT", 6379, int)
REDIS_SLAVE_HOST = get_env("REDIS_SLAVE_HOST", "redis-slave") REDIS_SLAVE_HOST = get_env("REDIS_SLAVE_HOST", "redis-slave")
REDIS_SLAVE_PORT = get_env("REDIS_SLAVE_PORT", 6379, int) REDIS_SLAVE_PORT = get_env("REDIS_SLAVE_PORT", 6379, int)
REDIS_CHANNLES_HOST = get_env("REDIS_CHANNLES_HOST", "redis-channels")
REDIS_CHANNLES_PORT = get_env("REDIS_CHANNLES_PORT", 6379, int)
REDIS_SLAVE_WAIT_TIMEOUT = get_env("REDIS_SLAVE_WAIT_TIMEOUT", 10000, int)
# Django Channels
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [(REDIS_CHANNLES_HOST, REDIS_CHANNLES_PORT)],
"capacity": 10000,
},
},
}
# Collection Cache # Collection Cache
REDIS_ADDRESS = f"redis://{REDIS_HOST}:{REDIS_PORT}/0" REDIS_ADDRESS = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
REDIS_READ_ONLY_ADDRESS = f"redis://{REDIS_SLAVE_HOST}:{REDIS_SLAVE_PORT}/0" REDIS_READ_ONLY_ADDRESS = f"redis://{REDIS_SLAVE_HOST}:{REDIS_SLAVE_PORT}/0"
AMOUNT_REPLICAS = get_env("AMOUNT_REPLICAS", 1, int)
CONNECTION_POOL_LIMIT = get_env("CONNECTION_POOL_LIMIT", 100, int) CONNECTION_POOL_LIMIT = get_env("CONNECTION_POOL_LIMIT", 100, int)
# Session backend # Session backend
@ -135,8 +120,6 @@ ENABLE_SAML = get_env("ENABLE_SAML", False, bool)
if ENABLE_SAML: if ENABLE_SAML:
INSTALLED_APPS += ["openslides.saml"] INSTALLED_APPS += ["openslides.saml"]
# TODO: More saml stuff...
# Controls if electronic voting (means non-analog polls) are enabled. # Controls if electronic voting (means non-analog polls) are enabled.
ENABLE_ELECTRONIC_VOTING = get_env("ENABLE_ELECTRONIC_VOTING", False, bool) ENABLE_ELECTRONIC_VOTING = get_env("ENABLE_ELECTRONIC_VOTING", False, bool)

View File

@ -15,7 +15,6 @@ class AgendaAppConfig(AppConfig):
from ..utils.access_permissions import required_user from ..utils.access_permissions import required_user
from ..utils.rest_api import router from ..utils.rest_api import router
from . import serializers # noqa from . import serializers # noqa
from .projector import register_projector_slides
from .signals import ( from .signals import (
get_permission_change_data, get_permission_change_data,
listen_to_related_object_post_delete, listen_to_related_object_post_delete,
@ -23,9 +22,6 @@ class AgendaAppConfig(AppConfig):
) )
from .views import ItemViewSet, ListOfSpeakersViewSet from .views import ItemViewSet, ListOfSpeakersViewSet
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
post_save.connect( post_save.connect(
listen_to_related_object_post_save, listen_to_related_object_post_save,

View File

@ -1,309 +0,0 @@
from collections import defaultdict
from typing import Any, Dict, List, Union
from ..users.projector import get_user_name
from ..utils.projector import (
ProjectorAllDataProvider,
ProjectorElementException,
get_config,
get_model,
register_projector_slide,
)
# Important: All functions have to be prune. This means, that thay can only
# access the data, that they get as argument and do not have any
# side effects.
async def get_sorted_agenda_items(
agenda_items: Dict[int, Dict[str, Any]]
) -> List[Dict[str, Any]]:
"""
Returns all sorted agenda items by id first and then weight, resulting in
ordered items, if some have the same weight.
"""
return sorted(
sorted(agenda_items.values(), key=lambda item: item["id"]),
key=lambda item: item["weight"],
)
async def get_flat_tree(
agenda_items: Dict[int, Dict[str, Any]], parent_id: int = 0
) -> List[Dict[str, Any]]:
"""
Build the item tree from all_data_provider.
Only build the tree from elements unterneath parent_id.
Returns a list of two element tuples where the first element is the item title
and the second a List with children as two element tuples.
"""
# Build a dict from an item_id to all its children
children: Dict[int, List[int]] = defaultdict(list)
for item in await get_sorted_agenda_items(agenda_items):
if item["type"] == 1: # only normal items
children[item["parent_id"] or 0].append(item["id"])
tree = []
def build_tree(item_ids: List[int], depth: int) -> None:
for item_id in item_ids:
item = agenda_items[item_id]
title_information = item["title_information"]
title_information["_agenda_item_number"] = item["item_number"]
tree.append(
{
"title_information": title_information,
"collection": item["content_object"]["collection"],
"depth": depth,
}
)
build_tree(children[item_id], depth + 1)
build_tree(children[parent_id], 0)
return tree
async def item_list_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Item list slide.
Returns all root items or all children of an item.
"""
# fetch all items, so they are cached:
all_agenda_items = await all_data_provider.get_collection("agenda/item")
only_main_items = element.get("only_main_items", True)
if only_main_items:
agenda_items = []
for item in await get_sorted_agenda_items(all_agenda_items):
if item["parent_id"] is None and item["type"] == 1:
title_information = item["title_information"]
title_information["_agenda_item_number"] = item["item_number"]
agenda_items.append(
{
"title_information": title_information,
"collection": item["content_object"]["collection"],
}
)
else:
agenda_items = await get_flat_tree(all_agenda_items)
return {"items": agenda_items}
async def list_of_speakers_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
List of speakers slide.
Returns all usernames, that are on the list of speaker of a slide.
"""
list_of_speakers_id = element.get("id")
if list_of_speakers_id is None:
raise ProjectorElementException("id is required for list of speakers slide")
return await get_list_of_speakers_slide_data(all_data_provider, list_of_speakers_id)
async def get_list_of_speakers_slide_data(
all_data_provider: ProjectorAllDataProvider, list_of_speakers_id: int
) -> Dict[str, Any]:
list_of_speakers = await get_model(
all_data_provider, "agenda/list-of-speakers", list_of_speakers_id
)
title_information = list_of_speakers["title_information"]
# try to get the agenda item for the content object (which must not exist)
content_object = await get_model(
all_data_provider,
list_of_speakers["content_object"]["collection"],
list_of_speakers["content_object"]["id"],
)
agenda_item_id = content_object.get("agenda_item_id")
if agenda_item_id is not None:
agenda_item = await all_data_provider.get("agenda/item", agenda_item_id)
if agenda_item is not None:
title_information["_agenda_item_number"] = agenda_item["item_number"]
# Partition speaker objects to waiting, current and finished
speakers_waiting = []
speakers_finished = []
current_speaker = None
for speaker in list_of_speakers["speakers"]:
user = await get_user_name(all_data_provider, speaker["user_id"])
formatted_speaker = {
"user": user,
"marked": speaker["marked"],
"point_of_order": speaker["point_of_order"],
"weight": speaker["weight"],
"end_time": speaker["end_time"],
}
if speaker["begin_time"] is None and speaker["end_time"] is None:
speakers_waiting.append(formatted_speaker)
elif speaker["begin_time"] is not None and speaker["end_time"] is None:
current_speaker = formatted_speaker
else:
speakers_finished.append(formatted_speaker)
# sort speakers
speakers_waiting = sorted(speakers_waiting, key=lambda s: s["weight"])
speakers_finished = sorted(speakers_finished, key=lambda s: s["end_time"])
number_of_last_speakers = await get_config(
all_data_provider, "agenda_show_last_speakers"
)
number_of_next_speakers = await get_config(
all_data_provider, "agenda_show_next_speakers"
)
if number_of_last_speakers == 0:
speakers_finished = []
else:
# Take the last speakers
speakers_finished = speakers_finished[-number_of_last_speakers:]
if number_of_next_speakers != -1:
speakers_waiting = speakers_waiting[:number_of_next_speakers]
return {
"waiting": speakers_waiting,
"current": current_speaker,
"finished": speakers_finished,
"content_object_collection": list_of_speakers["content_object"]["collection"],
"title_information": title_information,
"closed": list_of_speakers["closed"],
}
async def get_current_list_of_speakers_id_for_projector(
all_data_provider: ProjectorAllDataProvider, projector: Dict[str, Any]
) -> Union[int, None]:
"""
Search for elements, that do have a list of speakers:
Try to get a model by the collection and id in the element. This
model needs to have a 'list_of_speakers_id'. This list of speakers
must exist. The first matching element is taken.
"""
elements = projector["elements"]
list_of_speakers_id = None
for element in elements:
if "id" not in element:
continue
collection = element["name"]
id = element["id"]
model = await all_data_provider.get(collection, id)
if model is None:
continue
if "list_of_speakers_id" not in model:
continue
list_of_speakers_id = model["list_of_speakers_id"]
los_exists = await all_data_provider.exists(
"agenda/list-of-speakers", list_of_speakers_id
)
if not los_exists:
continue
break
return list_of_speakers_id
async def get_reference_projector(
all_data_provider: ProjectorAllDataProvider, projector_id: int
) -> Dict[str, Any]:
"""
Returns the reference projector to the given projector (by id)
"""
this_projector = await get_model(all_data_provider, "core/projector", projector_id)
reference_projector_id = this_projector["reference_projector_id"] or projector_id
return await get_model(all_data_provider, "core/projector", reference_projector_id)
async def current_list_of_speakers_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
The current list of speakers slide. Creates the data for the given projector.
"""
reference_projector = await get_reference_projector(all_data_provider, projector_id)
list_of_speakers_id = await get_current_list_of_speakers_id_for_projector(
all_data_provider, reference_projector
)
if list_of_speakers_id is None: # no element found
return {}
return await get_list_of_speakers_slide_data(all_data_provider, list_of_speakers_id)
async def current_speaker_chyron_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Returns the username for the current speaker.
"""
# get projector for color information
projector = await get_model(all_data_provider, "core/projector", projector_id)
slide_data = {
"background_color": projector["chyron_background_color"],
"font_color": projector["chyron_font_color"],
}
reference_projector = await get_reference_projector(all_data_provider, projector_id)
list_of_speakers_id = await get_current_list_of_speakers_id_for_projector(
all_data_provider, reference_projector
)
if list_of_speakers_id is None: # no element found
return slide_data
# get list of speakers to search current speaker
list_of_speakers = await get_model(
all_data_provider, "agenda/list-of-speakers", list_of_speakers_id
)
# find current speaker
current_speaker = None
for speaker in list_of_speakers["speakers"]:
if speaker["begin_time"] is not None and speaker["end_time"] is None:
current_speaker = await get_user_name(all_data_provider, speaker["user_id"])
break
if current_speaker is not None:
slide_data["current_speaker"] = current_speaker
return slide_data
def register_projector_slides() -> None:
register_projector_slide("agenda/item-list", item_list_slide)
register_projector_slide("agenda/list-of-speakers", list_of_speakers_slide)
register_projector_slide(
"agenda/current-list-of-speakers", current_list_of_speakers_slide
)
register_projector_slide(
"agenda/current-list-of-speakers-overlay", current_list_of_speakers_slide
)
register_projector_slide(
"agenda/current-speaker-chyron", current_speaker_chyron_slide
)

View File

@ -13,7 +13,6 @@ class AssignmentsAppConfig(AppConfig):
from ..utils.access_permissions import required_user from ..utils.access_permissions import required_user
from ..utils.rest_api import router from ..utils.rest_api import router
from . import serializers # noqa from . import serializers # noqa
from .projector import register_projector_slides
from .signals import get_permission_change_data from .signals import get_permission_change_data
from .views import ( from .views import (
AssignmentOptionViewSet, AssignmentOptionViewSet,
@ -22,9 +21,6 @@ class AssignmentsAppConfig(AppConfig):
AssignmentVoteViewSet, AssignmentVoteViewSet,
) )
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
permission_change.connect( permission_change.connect(
get_permission_change_data, get_permission_change_data,

View File

@ -1,113 +0,0 @@
from typing import Any, Dict, List
from ..users.projector import get_user_name
from ..utils.projector import (
ProjectorAllDataProvider,
get_model,
get_models,
register_projector_slide,
)
from .models import AssignmentPoll
async def assignment_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Assignment slide.
"""
assignment = await get_model(
all_data_provider, "assignments/assignment", element.get("id")
)
assignment_related_users: List[Dict[str, Any]] = [
{"user": await get_user_name(all_data_provider, aru["user_id"])}
for aru in sorted(
assignment["assignment_related_users"], key=lambda aru: aru["weight"]
)
]
return {
"title": assignment["title"],
"phase": assignment["phase"],
"open_posts": assignment["open_posts"],
"description": assignment["description"],
"assignment_related_users": assignment_related_users,
"number_poll_candidates": assignment["number_poll_candidates"],
}
async def assignment_poll_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Poll slide.
"""
poll = await get_model(
all_data_provider, "assignments/assignment-poll", element.get("id")
)
assignment = await get_model(
all_data_provider, "assignments/assignment", poll["assignment_id"]
)
poll_data = {
key: poll[key]
for key in (
"title",
"type",
"pollmethod",
"min_votes_amount",
"max_votes_amount",
"description",
"state",
"onehundred_percent_base",
"majority_method",
)
}
# Add options:
poll_data["options"] = []
options = await get_models(
all_data_provider, "assignments/assignment-option", poll["options_id"]
)
for option in sorted(options, key=lambda option: option["weight"]):
option_data: Dict[str, Any] = {
"user": {
"short_name": await get_user_name(all_data_provider, option["user_id"])
}
}
if poll["state"] == AssignmentPoll.STATE_PUBLISHED:
option_data["yes"] = float(option["yes"])
option_data["no"] = float(option["no"])
option_data["abstain"] = float(option["abstain"])
poll_data["options"].append(option_data)
if poll["state"] == AssignmentPoll.STATE_PUBLISHED:
poll_data["amount_global_yes"] = (
float(poll["amount_global_yes"]) if poll["amount_global_yes"] else None
)
poll_data["amount_global_no"] = (
float(poll["amount_global_no"]) if poll["amount_global_no"] else None
)
poll_data["amount_global_abstain"] = (
float(poll["amount_global_abstain"])
if poll["amount_global_abstain"]
else None
)
poll_data["votesvalid"] = float(poll["votesvalid"])
poll_data["votesinvalid"] = float(poll["votesinvalid"])
poll_data["votescast"] = float(poll["votescast"])
return {
"assignment": {"title": assignment["title"]},
"poll": poll_data,
}
def register_projector_slides() -> None:
register_projector_slide("assignments/assignment", assignment_slide)
register_projector_slide("assignments/assignment-poll", assignment_poll_slide)

View File

@ -16,11 +16,9 @@ class CoreAppConfig(AppConfig):
def ready(self): def ready(self):
# Import all required stuff. # Import all required stuff.
# Let all client websocket message register # Let all client websocket message register
from ..utils import websocket_client_messages # noqa
from ..utils.rest_api import router from ..utils.rest_api import router
from . import serializers # noqa from . import serializers # noqa
from .config import config from .config import config
from .projector import register_projector_slides
from .signals import ( from .signals import (
autoupdate_for_many_to_many_relations, autoupdate_for_many_to_many_relations,
cleanup_unused_permissions, cleanup_unused_permissions,
@ -41,9 +39,6 @@ class CoreAppConfig(AppConfig):
# Collect all config variables before getting the constants. # Collect all config variables before getting the constants.
config.collect_config_variables_from_apps() config.collect_config_variables_from_apps()
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
post_permission_creation.connect( post_permission_creation.connect(
delete_django_app_permissions, dispatch_uid="delete_django_app_permissions" delete_django_app_permissions, dispatch_uid="delete_django_app_permissions"
@ -126,6 +121,7 @@ class CoreAppConfig(AppConfig):
# Client settings # Client settings
client_settings_keys = [ client_settings_keys = [
"AUTOUPDATE_DELAY",
"PRIORITIZED_GROUP_IDS", "PRIORITIZED_GROUP_IDS",
"PING_INTERVAL", "PING_INTERVAL",
"PING_TIMEOUT", "PING_TIMEOUT",

View File

@ -1,68 +0,0 @@
from typing import Any, Dict
from ..utils.projector import (
ProjectorAllDataProvider,
get_config,
get_model,
register_projector_slide,
)
async def countdown_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Countdown slide.
Returns the full_data of the countdown element.
element = {
name: 'core/countdown',
id: 5, # Countdown ID
}
"""
countdown = await get_model(all_data_provider, "core/countdown", element.get("id"))
return {
"description": countdown["description"],
"running": countdown["running"],
"countdown_time": countdown["countdown_time"],
"warning_time": await get_config(
all_data_provider, "agenda_countdown_warning_time"
),
}
async def message_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Message slide.
Returns the full_data of the message element.
element = {
name: 'core/projector-message',
id: 5, # ProjectorMessage ID
}
"""
return await get_model(
all_data_provider, "core/projector-message", element.get("id")
)
async def clock_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
return {}
def register_projector_slides() -> None:
register_projector_slide("core/countdown", countdown_slide)
register_projector_slide("core/projector-message", message_slide)
register_projector_slide("core/clock", clock_slide)

View File

@ -1,7 +1,6 @@
from typing import Any from typing import Any
from ..core.config import config from ..core.config import config
from ..utils.projector import projector_slides
from ..utils.rest_api import ( from ..utils.rest_api import (
Field, Field,
IdPrimaryKeyRelatedField, IdPrimaryKeyRelatedField,
@ -62,10 +61,6 @@ def elements_validator(value: Any) -> None:
raise ValidationError( raise ValidationError(
{"detail": "Every dictionary must have a key 'name'."} {"detail": "Every dictionary must have a key 'name'."}
) )
if element["name"] not in projector_slides:
raise ValidationError(
{"detail": "Unknown projector element {0}.", "args": [element["name"]]}
)
def elements_array_validator(value: Any) -> None: def elements_array_validator(value: Any) -> None:

View File

@ -4,7 +4,8 @@ from . import views
urlpatterns = [ urlpatterns = [
url(r"^servertime/$", views.ServerTime.as_view(), name="core_servertime"), url(r"^servertime/$", views.ServertimeView.as_view(), name="core_servertime"),
url(r"^constants/$", views.ConstantsView.as_view(), name="core_constants"),
url(r"^version/$", views.VersionView.as_view(), name="core_version"), url(r"^version/$", views.VersionView.as_view(), name="core_version"),
url( url(
r"^history/information/$", r"^history/information/$",

View File

@ -22,6 +22,7 @@ from ..utils.arguments import arguments
from ..utils.auth import GROUP_ADMIN_PK, anonymous_is_enabled, has_perm, in_some_groups from ..utils.auth import GROUP_ADMIN_PK, anonymous_is_enabled, has_perm, in_some_groups
from ..utils.autoupdate import inform_changed_data from ..utils.autoupdate import inform_changed_data
from ..utils.cache import element_cache from ..utils.cache import element_cache
from ..utils.constants import get_constants
from ..utils.plugins import ( from ..utils.plugins import (
get_plugin_description, get_plugin_description,
get_plugin_license, get_plugin_license,
@ -569,7 +570,7 @@ class CountdownViewSet(ModelViewSet):
# Special API views # Special API views
class ServerTime(utils_views.APIView): class ServertimeView(utils_views.APIView):
""" """
Returns the server time as UNIX timestamp. Returns the server time as UNIX timestamp.
""" """
@ -580,6 +581,19 @@ class ServerTime(utils_views.APIView):
return now().timestamp() return now().timestamp()
class ConstantsView(utils_views.APIView):
"""
Returns the server time as UNIX timestamp.
"""
http_method_names = ["get"]
def get_context_data(self, **context):
if not self.request.user.is_authenticated and not anonymous_is_enabled():
self.permission_denied(self.request)
return get_constants()
class VersionView(utils_views.APIView): class VersionView(utils_views.APIView):
""" """
Returns a dictionary with the OpenSlides version and the version of all Returns a dictionary with the OpenSlides version and the version of all

View File

@ -16,7 +16,6 @@ INSTALLED_APPS = [
"django.contrib.sessions", "django.contrib.sessions",
"django.contrib.staticfiles", "django.contrib.staticfiles",
"rest_framework", "rest_framework",
"channels",
"openslides.agenda", "openslides.agenda",
"openslides.topics", "openslides.topics",
"openslides.motions", "openslides.motions",
@ -122,13 +121,5 @@ PASSWORD_HASHERS = [
MEDIA_URL = "/media/" MEDIA_URL = "/media/"
# Django Channels
# http://channels.readthedocs.io/en/latest/
ASGI_APPLICATION = "openslides.routing.application"
CHANNEL_LAYERS = {"default": {"BACKEND": "channels.layers.InMemoryChannelLayer"}}
# Enable updating the last_login field for users on every login. # Enable updating the last_login field for users on every login.
ENABLE_LAST_LOGIN_FIELD = False ENABLE_LAST_LOGIN_FIELD = False

View File

@ -11,9 +11,7 @@ class MediafilesAppConfig(AppConfig):
# Import all required stuff. # Import all required stuff.
from openslides.core.signals import permission_change from openslides.core.signals import permission_change
from openslides.utils.rest_api import router from openslides.utils.rest_api import router
from . import serializers # noqa from . import serializers # noqa
from .projector import register_projector_slides
from .signals import get_permission_change_data from .signals import get_permission_change_data
from .views import MediafileViewSet from .views import MediafileViewSet
@ -25,9 +23,6 @@ class MediafilesAppConfig(AppConfig):
"The MEDIA_URL setting must start and end with a slash" "The MEDIA_URL setting must start and end with a slash"
) )
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
permission_change.connect( permission_change.connect(
get_permission_change_data, get_permission_change_data,

View File

@ -1,29 +0,0 @@
from typing import Any, Dict
from ..utils.projector import (
ProjectorAllDataProvider,
get_model,
register_projector_slide,
)
async def mediafile_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Slide for Mediafile.
"""
mediafile = await get_model(
all_data_provider, "mediafiles/mediafile", element.get("id")
)
return {
"path": mediafile["path"],
"mimetype": mediafile["mimetype"],
"media_url_prefix": mediafile["media_url_prefix"],
}
def register_projector_slides() -> None:
register_projector_slide("mediafiles/mediafile", mediafile_slide)

View File

@ -12,10 +12,8 @@ class MotionsAppConfig(AppConfig):
# Import all required stuff. # Import all required stuff.
from openslides.core.signals import permission_change from openslides.core.signals import permission_change
from openslides.utils.rest_api import router from openslides.utils.rest_api import router
from ..utils.access_permissions import required_user from ..utils.access_permissions import required_user
from . import serializers # noqa from . import serializers # noqa
from .projector import register_projector_slides
from .signals import create_builtin_workflows, get_permission_change_data from .signals import create_builtin_workflows, get_permission_change_data
from .views import ( from .views import (
CategoryViewSet, CategoryViewSet,
@ -31,9 +29,6 @@ class MotionsAppConfig(AppConfig):
WorkflowViewSet, WorkflowViewSet,
) )
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
post_migrate.connect( post_migrate.connect(
create_builtin_workflows, dispatch_uid="motion_create_builtin_workflows" create_builtin_workflows, dispatch_uid="motion_create_builtin_workflows"

View File

@ -1,437 +0,0 @@
import re
from typing import Any, Dict, List, Optional
from ..users.projector import get_user_name
from ..utils.projector import (
ProjectorAllDataProvider,
ProjectorElementException,
get_config,
get_model,
register_projector_slide,
)
from .models import MotionPoll
motion_placeholder_regex = re.compile(r"\[motion:(\d+)\]")
async def get_state(
all_data_provider: ProjectorAllDataProvider,
motion: Dict[str, Any],
state_id_key: str,
) -> Dict[str, Any]:
"""
Returns a state element from one motion. Raises an error if the state does not exist.
"""
state = await all_data_provider.get("motions/state", motion[state_id_key])
if state is None:
raise ProjectorElementException(
f"motion {motion['id']} can not be on the state with id {motion[state_id_key]}"
)
return state
async def get_amendment_merge_into_motion_diff(all_data_provider, amendment):
"""
HINT: This implementation should be consistent to showInDiffView() in ViewMotionAmendedParagraph.ts
"""
if amendment["state_id"] is None:
return 0
state = await get_state(all_data_provider, amendment, "state_id")
if state["merge_amendment_into_final"] == -1:
return 0
if state["merge_amendment_into_final"] == 1:
return 1
if amendment["recommendation_id"] is None:
return 0
recommendation = await get_state(all_data_provider, amendment, "recommendation_id")
if recommendation["merge_amendment_into_final"] == 1:
return 1
return 0
async def get_amendment_merge_into_motion_final(all_data_provider, amendment):
"""
HINT: This implementation should be consistent to showInFinalView() in ViewMotionAmendedParagraph.ts
"""
if amendment["state_id"] is None:
return 0
state = await get_state(all_data_provider, amendment, "state_id")
if state["merge_amendment_into_final"] == 1:
return 1
return 0
async def get_amendments_for_motion(motion, all_data_provider):
amendment_data = []
for amendment_id in motion["amendments_id"]:
amendment = await all_data_provider.get("motions/motion", amendment_id)
merge_amendment_into_final = await get_amendment_merge_into_motion_final(
all_data_provider, amendment
)
merge_amendment_into_diff = await get_amendment_merge_into_motion_diff(
all_data_provider, amendment
)
# Add change recommendations to the amendments:
change_recommendations = [] # type: ignore
for change_recommendation_id in amendment["change_recommendations_id"]:
cr = await get_model(
all_data_provider,
"motions/motion-change-recommendation",
change_recommendation_id,
)
if cr is not None and not cr["internal"] and not cr["rejected"]:
change_recommendations.append(cr)
amendment_data.append(
{
"id": amendment["id"],
"identifier": amendment["identifier"],
"title": amendment["title"],
"amendment_paragraphs": amendment["amendment_paragraphs"],
"change_recommendations": change_recommendations,
"merge_amendment_into_diff": merge_amendment_into_diff,
"merge_amendment_into_final": merge_amendment_into_final,
}
)
return amendment_data
async def get_amendment_base_motion(amendment, all_data_provider):
motion = await get_model(
all_data_provider, "motions/motion", amendment.get("parent_id")
)
return {
"identifier": motion["identifier"],
"title": motion["title"],
"text": motion["text"],
}
async def get_amendment_base_statute(amendment, all_data_provider):
statute = await get_model(
all_data_provider,
"motions/statute-paragraph",
amendment.get("statute_paragraph_id"),
)
return {"title": statute["title"], "text": statute["text"]}
async def extend_reference_motion_dict(
all_data_provider: ProjectorAllDataProvider,
recommendation: Optional[str],
referenced_motions: Dict[int, Dict[str, str]],
) -> None:
"""
Extends a dict of motion ids mapped to their title information.
The client can replace the placeholders in the recommendation correctly.
"""
if recommendation is None:
return
# Collect all meantioned motions via [motion:<id>]
referenced_ids = [
int(id) for id in motion_placeholder_regex.findall(recommendation)
]
for id in referenced_ids:
# Put every referenced motion into the referenced_motions dict
referenced_motion = await all_data_provider.get("motions/motion", id)
if id not in referenced_motions and referenced_motion is not None:
referenced_motions[id] = {
"title": referenced_motion["title"],
"identifier": referenced_motion["identifier"],
}
async def motion_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Motion slide.
The returned dict can contain the following fields:
* identifier
* title
* text
* submitters
* amendment_paragraphs
* is_child
* show_meta_box
* show_referring_motions
* reason
* modified_final_version
* recommendation
* recommendation_extension
* recommender
* change_recommendations
"""
# Get motion
mode = element.get(
"mode", await get_config(all_data_provider, "motions_recommendation_text_mode")
)
# populate cache:
motion = await get_model(all_data_provider, "motions/motion", element.get("id"))
# Add submitters
submitters = [
await get_user_name(all_data_provider, submitter["user_id"])
for submitter in sorted(
motion["submitters"], key=lambda submitter: submitter["weight"]
)
]
# Get some needed config values
show_meta_box = not await get_config(
all_data_provider, "motions_disable_sidebox_on_projector"
)
show_referring_motions = not await get_config(
all_data_provider, "motions_hide_referring_motions"
)
line_length = await get_config(all_data_provider, "motions_line_length")
line_numbering_mode = await get_config(
all_data_provider, "motions_default_line_numbering"
)
motions_preamble = await get_config(all_data_provider, "motions_preamble")
# Query all change-recommendation and amendment related things.
amendments = [] # type: ignore
base_motion = None
base_statute = None
if motion["statute_paragraph_id"]:
base_statute = await get_amendment_base_statute(motion, all_data_provider)
elif motion["parent_id"] is not None and motion["amendment_paragraphs"]:
base_motion = await get_amendment_base_motion(motion, all_data_provider)
else:
amendments = await get_amendments_for_motion(motion, all_data_provider)
change_recommendations = [] # type: ignore
for change_recommendation_id in motion["change_recommendations_id"]:
cr = await get_model(
all_data_provider,
"motions/motion-change-recommendation",
change_recommendation_id,
)
if cr is not None and not cr["internal"]:
change_recommendations.append(cr)
# The base return value. More fields will get added below.
return_value = {
"identifier": motion["identifier"],
"title": motion["title"],
"submitters": submitters,
"preamble": motions_preamble,
"amendment_paragraphs": motion["amendment_paragraphs"],
"base_motion": base_motion,
"base_statute": base_statute,
"is_child": bool(motion["parent_id"]),
"show_meta_box": show_meta_box,
"show_referring_motions": show_referring_motions,
"change_recommendations": change_recommendations,
"amendments": amendments,
"line_length": line_length,
"line_numbering_mode": line_numbering_mode,
}
if not await get_config(all_data_provider, "motions_disable_text_on_projector"):
return_value["text"] = motion["text"]
if not await get_config(all_data_provider, "motions_disable_reason_on_projector"):
return_value["reason"] = motion["reason"]
if mode == "final":
return_value["modified_final_version"] = motion["modified_final_version"]
# Add recommendation, if enabled in config (and the motion has one)
if (
not await get_config(
all_data_provider, "motions_disable_recommendation_on_projector"
)
and motion["recommendation_id"]
):
recommendation_state = await get_state(
all_data_provider, motion, "recommendation_id"
)
return_value["recommendation"] = recommendation_state["recommendation_label"]
if recommendation_state["show_recommendation_extension_field"]:
recommendation_extension = motion["recommendation_extension"]
# All title information for referenced motions in the recommendation
referenced_motions: Dict[int, Dict[str, str]] = {}
await extend_reference_motion_dict(
all_data_provider, recommendation_extension, referenced_motions
)
return_value["recommendation_extension"] = recommendation_extension
return_value["referenced_motions"] = referenced_motions
if motion["statute_paragraph_id"]:
return_value["recommender"] = await get_config(
all_data_provider, "motions_statute_recommendations_by"
)
else:
return_value["recommender"] = await get_config(
all_data_provider, "motions_recommendations_by"
)
if show_referring_motions:
# Add recommendation-referencing motions
return_value[
"recommendation_referencing_motions"
] = await get_recommendation_referencing_motions(
all_data_provider, motion["id"]
)
return return_value
async def get_recommendation_referencing_motions(
all_data_provider: ProjectorAllDataProvider, motion_id: int
) -> Optional[List[Dict[str, Any]]]:
"""
Returns all title information for motions, that are referencing
the given motion (by id) in their recommendation. If there are no
motions, None is returned (instead of []).
"""
recommendation_referencing_motions = []
all_motions = await all_data_provider.get_collection("motions/motion")
for motion in all_motions.values():
# Motion must have a recommendation and a recommendaiton extension
if not motion["recommendation_id"] or not motion["recommendation_extension"]:
continue
# The recommendation must allow the extension field (there might be left-overs
# in a motions recommendation extension..)
recommendation = await get_state(all_data_provider, motion, "recommendation_id")
if not recommendation["show_recommendation_extension_field"]:
continue
# Find referenced motion ids
referenced_ids = [
int(id)
for id in motion_placeholder_regex.findall(
motion["recommendation_extension"]
)
]
# if one of the referenced ids is the given motion, add the current motion.
if motion_id in referenced_ids:
recommendation_referencing_motions.append(
{"title": motion["title"], "identifier": motion["identifier"]}
)
return recommendation_referencing_motions or None
async def motion_block_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Motion block slide.
"""
motion_block = await get_model(
all_data_provider, "motions/motion-block", element.get("id")
)
# All motions in this motion block
motions = []
# All title information for referenced motions in the recommendation
referenced_motions: Dict[int, Dict[str, str]] = {}
# iterate motions.
for motion_id in motion_block["motions_id"]:
motion = await all_data_provider.get("motions/motion", motion_id)
# primarily to please mypy, should theoretically not happen
if motion is None:
raise RuntimeError(
f"motion {motion_id} of block {element.get('id')} could not be found"
)
motion_object = {
"title": motion["title"],
"identifier": motion["identifier"],
}
recommendation_id = motion["recommendation_id"]
if recommendation_id is not None:
recommendation = await get_state(
all_data_provider, motion, "recommendation_id"
)
motion_object["recommendation"] = {
"name": recommendation["recommendation_label"],
"css_class": recommendation["css_class"],
}
if recommendation["show_recommendation_extension_field"]:
recommendation_extension = motion["recommendation_extension"]
await extend_reference_motion_dict(
all_data_provider, recommendation_extension, referenced_motions
)
motion_object["recommendation_extension"] = recommendation_extension
motions.append(motion_object)
return {
"title": motion_block["title"],
"motions": motions,
"referenced_motions": referenced_motions,
}
async def motion_poll_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Poll slide.
"""
poll = await get_model(all_data_provider, "motions/motion-poll", element.get("id"))
motion = await get_model(all_data_provider, "motions/motion", poll["motion_id"])
poll_data = {
key: poll[key]
for key in (
"title",
"type",
"pollmethod",
"state",
"onehundred_percent_base",
"majority_method",
)
}
if poll["state"] == MotionPoll.STATE_PUBLISHED:
option = await get_model(
all_data_provider, "motions/motion-option", poll["options_id"][0]
) # there can only be exactly one option
poll_data["options"] = [
{
"yes": float(option["yes"]),
"no": float(option["no"]),
"abstain": float(option["abstain"]),
}
]
poll_data["votesvalid"] = poll["votesvalid"]
poll_data["votesinvalid"] = poll["votesinvalid"]
poll_data["votescast"] = poll["votescast"]
return {
"motion": {"title": motion["title"], "identifier": motion["identifier"]},
"poll": poll_data,
}
def register_projector_slides() -> None:
register_projector_slide("motions/motion", motion_slide)
register_projector_slide("motions/motion-block", motion_block_slide)
register_projector_slide("motions/motion-poll", motion_poll_slide)

View File

@ -1,15 +0,0 @@
from channels.routing import ProtocolTypeRouter, URLRouter
from django.conf.urls import url
from openslides.utils.consumers import CloseConsumer, SiteConsumer
from openslides.utils.middleware import AuthMiddlewareStack
application = ProtocolTypeRouter(
{
# WebSocket chat handler
"websocket": AuthMiddlewareStack(
URLRouter([url(r"^ws/$", SiteConsumer), url(".*", CloseConsumer)])
)
}
)

View File

@ -11,13 +11,9 @@ class TopicsAppConfig(AppConfig):
from ..utils.rest_api import router from ..utils.rest_api import router
from . import serializers # noqa from . import serializers # noqa
from .projector import register_projector_slides
from .signals import get_permission_change_data from .signals import get_permission_change_data
from .views import TopicViewSet from .views import TopicViewSet
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
permission_change.connect( permission_change.connect(
get_permission_change_data, dispatch_uid="topics_get_permission_change_data" get_permission_change_data, dispatch_uid="topics_get_permission_change_data"

View File

@ -1,32 +0,0 @@
from typing import Any, Dict
from ..utils.projector import (
ProjectorAllDataProvider,
get_model,
register_projector_slide,
)
async def topic_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
Topic slide.
The returned dict can contain the following fields:
* title
* text
"""
topic = await get_model(all_data_provider, "topics/topic", element.get("id"))
item = await get_model(all_data_provider, "agenda/item", topic["agenda_item_id"])
return {
"title": topic["title"],
"text": topic["text"],
"item_number": item["item_number"],
}
def register_projector_slides() -> None:
register_projector_slide("topics/topic", topic_slide)

View File

@ -15,13 +15,9 @@ class UsersAppConfig(AppConfig):
from ..core.signals import permission_change, post_permission_creation from ..core.signals import permission_change, post_permission_creation
from ..utils.rest_api import router from ..utils.rest_api import router
from . import serializers # noqa from . import serializers # noqa
from .projector import register_projector_slides
from .signals import create_builtin_groups_and_admin, get_permission_change_data from .signals import create_builtin_groups_and_admin, get_permission_change_data
from .views import GroupViewSet, PersonalNoteViewSet, UserViewSet from .views import GroupViewSet, PersonalNoteViewSet, UserViewSet
# Define projector elements.
register_projector_slides()
# Connect signals. # Connect signals.
post_permission_creation.connect( post_permission_creation.connect(
create_builtin_groups_and_admin, create_builtin_groups_and_admin,

View File

@ -1,44 +0,0 @@
from typing import Any, Dict, List, Optional
from ..utils.projector import (
ProjectorAllDataProvider,
get_model,
register_projector_slide,
)
async def user_slide(
all_data_provider: ProjectorAllDataProvider,
element: Dict[str, Any],
projector_id: int,
) -> Dict[str, Any]:
"""
User slide.
The returned dict can contain the following fields:
* user
"""
return {"user": await get_user_name(all_data_provider, element.get("id"))}
async def get_user_name(
all_data_provider: ProjectorAllDataProvider, user_id: Optional[int]
) -> str:
"""
Returns the short name for an user_id.
"""
user = await get_model(all_data_provider, "users/user", user_id)
name_parts: List[str] = []
for name_part in ("title", "first_name", "last_name"):
if user[name_part]:
name_parts.append(user[name_part])
if not name_parts:
name_parts.append(user["username"])
if user["structure_level"]:
name_parts.append(f"({user['structure_level']})")
return " ".join(name_parts)
def register_projector_slides() -> None:
register_projector_slide("users/user", user_slide)

View File

@ -4,13 +4,12 @@ from collections import defaultdict
from typing import Any, Dict, Iterable, List, Optional, Tuple, Union from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
from asgiref.sync import async_to_sync from asgiref.sync import async_to_sync
from channels.layers import get_channel_layer
from django.db.models import Model from django.db.models import Model
from mypy_extensions import TypedDict from mypy_extensions import TypedDict
from .auth import UserDoesNotExist from .auth import UserDoesNotExist
from .cache import ChangeIdTooLowError, element_cache, get_element_id from .cache import ChangeIdTooLowError, element_cache, get_element_id
from .projector import get_projector_data from .stream import stream
from .timing import Timing from .timing import Timing
from .utils import get_model_from_collection_string, is_iterable, split_element_id from .utils import get_model_from_collection_string, is_iterable, split_element_id
@ -112,8 +111,7 @@ class AutoupdateBundle:
save_history(self.element_iterator) save_history(self.element_iterator)
# Update cache and send autoupdate using async code. # Update cache and send autoupdate using async code.
change_id = async_to_sync(self.dispatch_autoupdate)() return async_to_sync(self.dispatch_autoupdate)()
return change_id
@property @property
def element_iterator(self) -> Iterable[AutoupdateElement]: def element_iterator(self) -> Iterable[AutoupdateElement]:
@ -121,7 +119,7 @@ class AutoupdateBundle:
for elements in self.autoupdate_elements.values(): for elements in self.autoupdate_elements.values():
yield from elements.values() yield from elements.values()
async def update_cache(self) -> int: async def get_data_for_cache(self) -> Dict[str, Optional[Dict[str, Any]]]:
""" """
Async helper function to update the cache. Async helper function to update the cache.
@ -136,7 +134,7 @@ class AutoupdateBundle:
"no_delete_on_restriction", False "no_delete_on_restriction", False
) )
cache_elements[element_id] = full_data cache_elements[element_id] = full_data
return await element_cache.change_elements(cache_elements) return cache_elements
async def dispatch_autoupdate(self) -> int: async def dispatch_autoupdate(self) -> int:
""" """
@ -145,25 +143,12 @@ class AutoupdateBundle:
Return the change_id Return the change_id
""" """
# Update cache # Update cache
change_id = await self.update_cache() cache_elements = await self.get_data_for_cache()
change_id = await element_cache.change_elements(cache_elements)
# Send autoupdate # Send autoupdate
channel_layer = get_channel_layer() autoupdate_payload = {"elements": cache_elements, "change_id": change_id}
await channel_layer.group_send( await stream.send("autoupdate", autoupdate_payload)
"autoupdate", {"type": "msg_new_change_id", "change_id": change_id}
)
# Send projector
projector_data = await get_projector_data()
channel_layer = get_channel_layer()
await channel_layer.group_send(
"projector",
{
"type": "msg_projector_data",
"data": projector_data,
"change_id": change_id,
},
)
return change_id return change_id
@ -286,14 +271,12 @@ class AutoupdateBundleMiddleware:
if status_ok or status_redirect: if status_ok or status_redirect:
change_id = bundle.done() change_id = bundle.done()
# inject the autoupdate, if there is an autoupdate and the status is # inject the change id, if there was an autoupdate and the response status is
# ok (and not redirect; redirects do not have a useful content) # ok (and not redirect; redirects do not have a useful content)
if change_id is not None and status_ok: if change_id is not None and status_ok:
user_id = request.user.pk or 0
# Inject the autoupdate in the response. # Inject the autoupdate in the response.
# The complete response body will be overwritten! # The complete response body will be overwritten!
_, autoupdate = async_to_sync(get_autoupdate_data)(change_id, user_id) content = {"change_id": change_id, "data": response.data}
content = {"autoupdate": autoupdate, "data": response.data}
# Note: autoupdate may be none on skipped ones (which should not happen # Note: autoupdate may be none on skipped ones (which should not happen
# since the user has made the request....) # since the user has made the request....)
response.content = json.dumps(content) response.content = json.dumps(content)

View File

@ -163,6 +163,8 @@ class ElementCache:
logger.info("Saving cache data into the cache...") logger.info("Saving cache data into the cache...")
await self.cache_provider.add_to_full_data(mapping) await self.cache_provider.add_to_full_data(mapping)
logger.info("Done saving the cache data.") logger.info("Done saving the cache data.")
await self.cache_provider.set_cache_ready()
logger.info("Done: Cache is ready now.")
def _build_cache_get_elementid_model_mapping( def _build_cache_get_elementid_model_mapping(
self, config_only: bool = False self, config_only: bool = False

View File

@ -8,11 +8,7 @@ from django.core.exceptions import ImproperlyConfigured
from typing_extensions import Protocol from typing_extensions import Protocol
from . import logging from . import logging
from .redis import ( from .redis import use_redis
read_only_redis_amount_replicas,
read_only_redis_wait_timeout,
use_redis,
)
from .schema_version import SchemaVersion from .schema_version import SchemaVersion
from .utils import split_element_id, str_dict_to_bytes from .utils import split_element_id, str_dict_to_bytes
@ -54,6 +50,9 @@ class ElementCacheProvider(Protocol):
async def data_exists(self) -> bool: async def data_exists(self) -> bool:
... ...
async def set_cache_ready(self) -> None:
...
async def get_all_data(self) -> Dict[bytes, bytes]: async def get_all_data(self) -> Dict[bytes, bytes]:
... ...
@ -127,6 +126,7 @@ class RedisCacheProvider:
full_data_cache_key: str = "full_data" full_data_cache_key: str = "full_data"
change_id_cache_key: str = "change_id" change_id_cache_key: str = "change_id"
schema_cache_key: str = "schema" schema_cache_key: str = "schema"
cache_ready_key: str = "cache_ready"
# All lua-scripts used by this provider. Every entry is a Tuple (str, bool) with the # All lua-scripts used by this provider. Every entry is a Tuple (str, bool) with the
# script and an ensure_cache-indicator. If the indicator is True, a short ensure_cache-script # script and an ensure_cache-indicator. If the indicator is True, a short ensure_cache-script
@ -325,6 +325,7 @@ class RedisCacheProvider:
""" """
async with get_connection() as redis: async with get_connection() as redis:
tr = redis.multi_exec() tr = redis.multi_exec()
tr.delete(self.cache_ready_key)
tr.delete(self.change_id_cache_key) tr.delete(self.change_id_cache_key)
tr.delete(self.full_data_cache_key) tr.delete(self.full_data_cache_key)
tr.hmset_dict(self.full_data_cache_key, data) tr.hmset_dict(self.full_data_cache_key, data)
@ -342,11 +343,16 @@ class RedisCacheProvider:
Returns True, when there is data in the cache. Returns True, when there is data in the cache.
""" """
async with get_connection(read_only=True) as redis: async with get_connection(read_only=True) as redis:
return await redis.exists(self.full_data_cache_key) and bool( return (await redis.get(self.cache_ready_key)) is not None
await redis.zrangebyscore( # return await redis.exists(self.full_data_cache_key) and bool(
self.change_id_cache_key, withscores=True, count=1, offset=0 # await redis.zrangebyscore(
) # self.change_id_cache_key, withscores=True, count=1, offset=0
) # )
# )
async def set_cache_ready(self) -> None:
async with get_connection(read_only=False) as redis:
await redis.set(self.cache_ready_key, "ok")
@ensure_cache_wrapper() @ensure_cache_wrapper()
async def get_all_data(self) -> Dict[bytes, bytes]: async def get_all_data(self) -> Dict[bytes, bytes]:
@ -495,7 +501,11 @@ class RedisCacheProvider:
async def get_schema_version(self) -> Optional[SchemaVersion]: async def get_schema_version(self) -> Optional[SchemaVersion]:
""" Retrieves the schema version of the cache or None, if not existent """ """ Retrieves the schema version of the cache or None, if not existent """
async with get_connection(read_only=True) as redis: async with get_connection(read_only=True) as redis:
try:
schema_version = await redis.hgetall(self.schema_cache_key) schema_version = await redis.hgetall(self.schema_cache_key)
except aioredis.errors.ReplyError:
await redis.delete(self.schema_cache_key)
return None
if not schema_version: if not schema_version:
return None return None
@ -543,15 +553,6 @@ class RedisCacheProvider:
raise CacheReset() raise CacheReset()
else: else:
raise e raise e
if not read_only and read_only_redis_amount_replicas is not None:
reported_amount = await redis.wait(
read_only_redis_amount_replicas, read_only_redis_wait_timeout
)
if reported_amount != read_only_redis_amount_replicas:
logger.warn(
f"WAIT reported {reported_amount} replicas of {read_only_redis_amount_replicas} "
+ f"requested after {read_only_redis_wait_timeout} ms!"
)
return result return result
async def _eval( async def _eval(
@ -584,6 +585,7 @@ class MemoryCacheProvider:
self.set_data_dicts() self.set_data_dicts()
def set_data_dicts(self) -> None: def set_data_dicts(self) -> None:
self.ready = False
self.full_data: Dict[str, str] = {} self.full_data: Dict[str, str] = {}
self.change_id_data: Dict[int, Set[str]] = {} self.change_id_data: Dict[int, Set[str]] = {}
self.locks: Dict[str, str] = {} self.locks: Dict[str, str] = {}
@ -594,6 +596,7 @@ class MemoryCacheProvider:
async def clear_cache(self) -> None: async def clear_cache(self) -> None:
self.set_data_dicts() self.set_data_dicts()
self.ready = False
async def reset_full_cache( async def reset_full_cache(
self, data: Dict[str, str], default_change_id: int self, data: Dict[str, str], default_change_id: int
@ -606,7 +609,11 @@ class MemoryCacheProvider:
self.full_data.update(data) self.full_data.update(data)
async def data_exists(self) -> bool: async def data_exists(self) -> bool:
return bool(self.full_data) and self.default_change_id >= 0 return self.ready
# return bool(self.full_data) and self.default_change_id >= 0
async def set_cache_ready(self) -> None:
self.ready = True
async def get_all_data(self) -> Dict[bytes, bytes]: async def get_all_data(self) -> Dict[bytes, bytes]:
return str_dict_to_bytes(self.full_data) return str_dict_to_bytes(self.full_data)

View File

@ -1,102 +0,0 @@
import asyncio
from asyncio import Task
from typing import Optional, cast
from django.conf import settings
from .autoupdate import get_autoupdate_data
from .cache import element_cache
from .websocket import ChangeIdTooHighException, ProtocollAsyncJsonWebsocketConsumer
AUTOUPDATE_DELAY = getattr(settings, "AUTOUPDATE_DELAY", None)
class ConsumerAutoupdateStrategy:
def __init__(self, consumer: ProtocollAsyncJsonWebsocketConsumer) -> None:
self.consumer = consumer
# client_change_id = None: unknown -> set on first autoupdate or request_change_id
# client_change_id is int: the change_id, the client knows about, so the next
# update must be from client_change_id+1 .. <next clange_id>
self.client_change_id: Optional[int] = None
self.next_send_time = None
self.timer_task_handle: Optional[Task[None]] = None
self.lock = asyncio.Lock()
async def request_change_id(
self, change_id: int, in_response: Optional[str] = None
) -> None:
"""
The change id is not inclusive, so the client is on change_id and wants
data from change_id+1 .. now
"""
# This resets the server side tracking of the client's change id.
async with self.lock:
await self.stop_timer()
max_change_id = await element_cache.get_current_change_id()
self.client_change_id = change_id
if self.client_change_id == max_change_id:
# The client is up-to-date, so nothing will be done
return None
if self.client_change_id > max_change_id:
message = (
f"Requested change_id {self.client_change_id} is higher than the "
+ f"highest change_id {max_change_id}."
)
raise ChangeIdTooHighException(message, in_response=in_response)
await self.send_autoupdate(in_response=in_response)
async def new_change_id(self, change_id: int) -> None:
async with self.lock:
if self.client_change_id is None:
# The -1 is to send this autoupdate as the first one to he client.
# Remember: the client_change_id is the change_id the client knows about
self.client_change_id = change_id - 1
if AUTOUPDATE_DELAY is None: # feature deactivated, send directly
await self.send_autoupdate()
elif self.timer_task_handle is None:
await self.start_timer()
async def get_running_loop(self) -> asyncio.AbstractEventLoop:
if hasattr(asyncio, "get_running_loop"):
return asyncio.get_running_loop() # type: ignore
else:
return asyncio.get_event_loop()
async def start_timer(self) -> None:
loop = await self.get_running_loop()
self.timer_task_handle = loop.create_task(self.timer_task())
async def stop_timer(self) -> None:
if self.timer_task_handle is not None:
self.timer_task_handle.cancel()
self.timer_task_handle = None
async def timer_task(self) -> None:
try:
await asyncio.sleep(AUTOUPDATE_DELAY)
except asyncio.CancelledError:
return
async with self.lock:
await self.send_autoupdate()
self.timer_task_handle = None
async def send_autoupdate(self, in_response: Optional[str] = None) -> None:
# here, 1 is added to the change_id, because the client_change_id is the id the client
# *knows* about -> the client needs client_change_id+1 since get_autoupdate_data is
# inclusive [change_id .. max_change_id].
max_change_id, autoupdate = await get_autoupdate_data(
cast(int, self.client_change_id) + 1, self.consumer.user_id
)
if autoupdate is not None:
self.client_change_id = max_change_id
# It will be send, so we can set the client_change_id
await self.consumer.send_json(
type="autoupdate", content=autoupdate, in_response=in_response
)

View File

@ -1,175 +0,0 @@
import time
from typing import Any, Dict, List, Optional, cast
from urllib.parse import parse_qs
from channels.generic.websocket import AsyncWebsocketConsumer
from . import logging
from .auth import UserDoesNotExist, async_anonymous_is_enabled
from .cache import element_cache
from .consumer_autoupdate_strategy import ConsumerAutoupdateStrategy
from .utils import get_worker_id
from .websocket import BaseWebsocketException, ProtocollAsyncJsonWebsocketConsumer
logger = logging.getLogger("openslides.websocket")
class SiteConsumer(ProtocollAsyncJsonWebsocketConsumer):
"""
Websocket Consumer for the site.
"""
groups = ["site"]
ID_COUNTER = 0
"""
ID counter for assigning each instance of this class an unique id.
"""
def __init__(self, *args: Any, **kwargs: Any) -> None:
self.projector_hash: Dict[int, int] = {}
SiteConsumer.ID_COUNTER += 1
self._id = get_worker_id() + "-" + str(SiteConsumer.ID_COUNTER)
self.autoupdate_strategy = ConsumerAutoupdateStrategy(self)
super().__init__(*args, **kwargs)
async def connect(self) -> None:
"""
A user connects to the site.
If it is an anonymous user and anonymous is disabled, the connection is closed.
Sends the startup data to the user.
"""
self.user_id = self.scope["user"]["id"]
self.connect_time = time.time()
# self.scope['user'] is the full_data dict of the user. For an
# anonymous user is it the dict {'id': 0}
change_id = None
if not await async_anonymous_is_enabled() and not self.user_id:
await self.accept() # workaround for #4009
await self.close()
logger.debug(f"connect: denied ({self._id})")
return
query_string = cast(
Dict[bytes, List[bytes]], parse_qs(self.scope["query_string"])
)
if b"change_id" in query_string:
try:
change_id = int(query_string[b"change_id"][0])
except ValueError:
await self.accept() # workaround for #4009
await self.close()
logger.debug(f"connect: wrong change id ({self._id})")
return
await self.accept()
if change_id is not None:
logger.debug(f"connect: change id {change_id} ({self._id})")
try:
await self.request_autoupdate(change_id)
except BaseWebsocketException as e:
await self.send_exception(e)
else:
logger.debug(f"connect: no change id ({self._id})")
await self.channel_layer.group_add("autoupdate", self.channel_name)
async def disconnect(self, close_code: int) -> None:
"""
A user disconnects. Remove it from autoupdate.
"""
await self.channel_layer.group_discard("autoupdate", self.channel_name)
active_seconds = int(time.time() - self.connect_time)
logger.debug(
f"disconnect code={close_code} active_secs={active_seconds} ({self._id})"
)
async def msg_new_change_id(self, event: Dict[str, Any]) -> None:
"""
Send changed or deleted elements to the user.
"""
change_id = event["change_id"]
try:
await self.autoupdate_strategy.new_change_id(change_id)
except UserDoesNotExist:
# Maybe the user was deleted, but a websocket connection is still open to the user.
# So we can close this connection and return.
await self.close()
async def msg_projector_data(self, event: Dict[str, Any]) -> None:
"""
The projector has changed.
"""
all_projector_data = event["data"]
change_id = event["change_id"]
projector_data: Dict[int, Dict[str, Any]] = {}
for projector_id in self.listen_projector_ids:
data = all_projector_data.get(projector_id, [])
new_hash = hash(str(data))
if new_hash != self.projector_hash.get(projector_id):
projector_data[projector_id] = data
self.projector_hash[projector_id] = new_hash
if projector_data:
await self.send_projector_data(projector_data, change_id=change_id)
async def msg_notify(self, event: Dict[str, Any]) -> None:
"""
Send a notify message to the user.
"""
item = event["incomming"]
users = item.get("users")
reply_channels = item.get("replyChannels")
if (
(isinstance(users, bool) and users)
or (isinstance(users, list) and self.user_id in users)
or (
isinstance(reply_channels, list) and self.channel_name in reply_channels
)
or (users is None and reply_channels is None)
):
item["senderChannelName"] = event["senderChannelName"]
item["senderUserId"] = event["senderUserId"]
await self.send_json(type="notify", content=item)
async def request_autoupdate(
self, change_id: int, in_response: Optional[str] = None
) -> None:
await self.autoupdate_strategy.request_change_id(
change_id, in_response=in_response
)
async def send_projector_data(
self,
data: Dict[int, Dict[str, Any]],
change_id: Optional[int] = None,
in_response: Optional[str] = None,
) -> None:
"""
Sends projector data to the consumer.
"""
if change_id is None:
change_id = await element_cache.get_current_change_id()
content = {"change_id": change_id, "data": data}
await self.send_json(type="projector", content=content, in_response=in_response)
class CloseConsumer(AsyncWebsocketConsumer):
""" Auto-closes the connection """
groups: List[str] = []
def __init__(self, args: Dict[str, Any], **kwargs: Any) -> None:
logger.info(f'Closing connection to unknown websocket url {args["path"]}')
async def connect(self) -> None:
await self.accept()
await self.close()

View File

@ -1,73 +0,0 @@
from typing import Any, Dict, Optional
from channels.auth import (
AuthMiddleware,
CookieMiddleware,
SessionMiddleware,
_get_user_session_key,
)
from django.conf import settings
from django.contrib.auth import BACKEND_SESSION_KEY, HASH_SESSION_KEY
from django.utils.crypto import constant_time_compare
from .cache import element_cache
class CollectionAuthMiddleware(AuthMiddleware):
"""
Like the channels AuthMiddleware but returns a user dict id instead of
a django Model as user.
"""
def populate_scope(self, scope: Dict[str, Any]) -> None:
# Make sure we have a session
if "session" not in scope:
raise ValueError(
"AuthMiddleware cannot find session in scope. SessionMiddleware must be above it."
)
# Add it to the scope if it's not there already
if "user" not in scope:
scope["user"] = {}
async def resolve_scope(self, scope: Dict[str, Any]) -> None:
scope["user"].update(await get_user(scope))
async def get_user(scope: Dict[str, Any]) -> Dict[str, Any]:
"""
Returns a user id from a channels-scope-session.
If no user is retrieved, return {'id': 0}.
"""
# This code is basicly from channels.auth:
# https://github.com/django/channels/blob/d5e81a78e96770127da79248349808b6ee6ec2a7/channels/auth.py#L16
if "session" not in scope:
raise ValueError(
"Cannot find session in scope. You should wrap your consumer in SessionMiddleware."
)
session = scope["session"]
user: Optional[Dict[str, Any]] = None
try:
user_id = _get_user_session_key(session)
backend_path = session[BACKEND_SESSION_KEY]
except KeyError:
pass
else:
if backend_path in settings.AUTHENTICATION_BACKENDS:
user = await element_cache.get_element_data("users/user", user_id)
if user:
# Verify the session
session_hash = session.get(HASH_SESSION_KEY)
session_hash_verified = session_hash and constant_time_compare(
session_hash, user["session_auth_hash"]
)
if not session_hash_verified:
session.flush()
user = None
return user or {"id": 0}
# Handy shortcut for applying all three layers at once
AuthMiddlewareStack = lambda inner: CookieMiddleware( # noqa
SessionMiddleware(CollectionAuthMiddleware(inner))
)

View File

@ -1,172 +0,0 @@
"""
General projector code.
Functions that handel the registration of projector elements and the rendering
of the data to present it on the projector.
"""
from collections import defaultdict
from typing import Any, Awaitable, Callable, Dict, List, Optional
from . import logging
from .cache import element_cache
logger = logging.getLogger(__name__)
class ProjectorElementException(Exception):
"""
Exception for errors in one element on the projector.
"""
class ProjectorAllDataProvider:
NON_EXISTENT_MARKER = object()
def __init__(self) -> None:
self.cache: Any = defaultdict(dict) # fuu you mypy
self.fetched_collection: Dict[str, bool] = {}
async def get(self, collection: str, id: int) -> Optional[Dict[str, Any]]:
cache_data = self.cache[collection].get(id)
if cache_data is None:
data: Any = await element_cache.get_element_data(collection, id)
if data is None:
data = ProjectorAllDataProvider.NON_EXISTENT_MARKER
self.cache[collection][id] = data
cache_data = self.cache[collection][id]
if cache_data == ProjectorAllDataProvider.NON_EXISTENT_MARKER:
return None
return cache_data
async def get_collection(self, collection: str) -> Dict[int, Dict[str, Any]]:
if not self.fetched_collection.get(collection, False):
collection_data = await element_cache.get_collection_data(collection)
self.cache[collection] = collection_data
self.fetched_collection[collection] = True
return self.cache[collection]
async def exists(self, collection: str, id: int) -> bool:
model = await self.get(collection, id)
return model is not None
ProjectorSlide = Callable[
[ProjectorAllDataProvider, Dict[str, Any], int], Awaitable[Dict[str, Any]]
]
projector_slides: Dict[str, ProjectorSlide] = {}
def register_projector_slide(name: str, slide: ProjectorSlide) -> None:
"""
Registers a projector slide.
Has to be called in the app.ready method.
"""
projector_slides[name] = slide
async def get_projector_data(
projector_ids: List[int] = None,
) -> Dict[int, List[Dict[str, Any]]]:
"""
Calculates and returns the data for one or all projectors.
The keys of the returned data are the projector ids as int. When converted
to json, the numbers will changed to strings like "1".
The data for each projector is a list of elements.
Each element is a dict where the keys are "elements", "data". "elements"
contains the projector elements. It is the same as the projector elements in
the database. "data" contains all necessary data to render the projector
element. The key can also be "error" if there is a generall error for the
slide. In this case the values "elements" and "data" are optional.
The returned value looks like this:
projector_data = {
1: [
{
"element": {
"name": "agenda/item-list",
},
"data": {
"items": []
},
},
],
}
"""
if projector_ids is None:
projector_ids = []
projector_data: Dict[int, List[Dict[str, Any]]] = {}
all_data_provider = ProjectorAllDataProvider()
projectors = await all_data_provider.get_collection("core/projector")
for projector_id, projector in projectors.items():
if projector_ids and projector_id not in projector_ids:
# only render the projector in question.
continue
if not projector["elements"]:
# Skip empty elements.
continue
projector_data[projector_id] = []
for element in projector["elements"]:
projector_slide = projector_slides[element["name"]]
try:
data = await projector_slide(all_data_provider, element, projector_id)
except ProjectorElementException as err:
data = {"error": str(err)}
projector_data[projector_id].append({"data": data, "element": element})
return projector_data
async def get_config(all_data_provider: ProjectorAllDataProvider, key: str) -> Any:
"""
Returns a config value from all_data_provider.
Triggers the cache early: It access `get_colelction` instead of `get`. It
allows for all successive queries for configs to be cached.
"""
from ..core.config import config
config_id = (await config.async_get_key_to_id())[key]
configs = await all_data_provider.get_collection(config.get_collection_string())
return configs[config_id]["value"]
async def get_model(
all_data_provider: ProjectorAllDataProvider, collection: str, id: Any
) -> Dict[str, Any]:
"""
Tries to get the model identified by the collection and id.
If the id is invalid or the model not found, ProjectorElementExceptions will be raised.
"""
if id is None:
raise ProjectorElementException(f"id is required for {collection} slide")
model = await all_data_provider.get(collection, id)
if model is None:
raise ProjectorElementException(f"{collection} with id {id} does not exist")
return model
async def get_models(
all_data_provider: ProjectorAllDataProvider, collection: str, ids: List[Any]
) -> List[Dict[str, Any]]:
"""
Tries to fetch all given models. Models are required to be all of the collection `collection`.
"""
logger.info(
f"Note: a call to `get_models` with {collection}/{ids}. This might be cache-intensive"
)
return [await get_model(all_data_provider, collection, id) for id in ids]

View File

@ -10,15 +10,13 @@ logger = logging.getLogger(__name__)
# Defaults # Defaults
use_redis = False use_redis = False
use_read_only_redis = False use_read_only_redis = False
read_only_redis_amount_replicas = None
read_only_redis_wait_timeout = None
try: try:
import aioredis import aioredis
except ImportError: except ImportError:
pass pass
else: else:
from .redis_connection_pool import ConnectionPool from .redis_connection_pool import ConnectionPool # type: ignore
# set use_redis to true, if there is a value for REDIS_ADDRESS in the settings # set use_redis to true, if there is a value for REDIS_ADDRESS in the settings
redis_address = getattr(settings, "REDIS_ADDRESS", "") redis_address = getattr(settings, "REDIS_ADDRESS", "")
@ -33,13 +31,6 @@ else:
if use_read_only_redis: if use_read_only_redis:
logger.info(f"Redis read only address {redis_read_only_address}") logger.info(f"Redis read only address {redis_read_only_address}")
read_only_pool = ConnectionPool({"address": redis_read_only_address}) read_only_pool = ConnectionPool({"address": redis_read_only_address})
read_only_redis_amount_replicas = getattr(settings, "AMOUNT_REPLICAS", 1)
logger.info(f"AMOUNT_REPLICAS={read_only_redis_amount_replicas}")
read_only_redis_wait_timeout = getattr(
settings, "REDIS_SLAVE_WAIT_TIMEOUT", 1000
)
logger.info(f"REDIS_SLAVE_WAIT_TIMEOUT={read_only_redis_wait_timeout}")
else: else:
logger.info("Redis is not configured.") logger.info("Redis is not configured.")

View File

@ -1,8 +1,10 @@
# type: ignore
import asyncio import asyncio
import sys
import types
from typing import Any, Dict, List, Optional from typing import Any, Dict, List, Optional
import aioredis import aioredis
from channels_redis.core import ConnectionPool as ChannelRedisConnectionPool
from django.conf import settings from django.conf import settings
from . import logging from . import logging
@ -13,6 +15,128 @@ connection_pool_limit = getattr(settings, "CONNECTION_POOL_LIMIT", 100)
logger.info(f"CONNECTION_POOL_LIMIT={connection_pool_limit}") logger.info(f"CONNECTION_POOL_LIMIT={connection_pool_limit}")
# Copied from https://github.com/django/channels_redis/blob/master/channels_redis/core.py
# and renamed..
AIOREDIS_VERSION = tuple(map(int, aioredis.__version__.split(".")))
def _wrap_close(loop, pool):
"""
Decorate an event loop's close method with our own.
"""
original_impl = loop.close
def _wrapper(self, *args, **kwargs):
# If the event loop was closed, there's nothing we can do anymore.
if not self.is_closed():
self.run_until_complete(pool.close_loop(self))
# Restore the original close() implementation after we're done.
self.close = original_impl
return self.close(*args, **kwargs)
loop.close = types.MethodType(_wrapper, loop)
class ChannelRedisConnectionPool:
"""
Connection pool manager for the channel layer.
It manages a set of connections for the given host specification and
taking into account asyncio event loops.
"""
def __init__(self, host):
self.host = host
self.conn_map = {}
self.in_use = {}
def _ensure_loop(self, loop):
"""
Get connection list for the specified loop.
"""
if loop is None:
loop = asyncio.get_event_loop()
if loop not in self.conn_map:
# Swap the loop's close method with our own so we get
# a chance to do some cleanup.
_wrap_close(loop, self)
self.conn_map[loop] = []
return self.conn_map[loop], loop
async def pop(self, loop=None):
"""
Get a connection for the given identifier and loop.
"""
conns, loop = self._ensure_loop(loop)
if not conns:
if sys.version_info >= (3, 8, 0) and AIOREDIS_VERSION >= (1, 3, 1):
conn = await aioredis.create_redis(**self.host)
else:
conn = await aioredis.create_redis(**self.host, loop=loop)
conns.append(conn)
conn = conns.pop()
if conn.closed:
conn = await self.pop(loop=loop)
return conn
self.in_use[conn] = loop
return conn
def push(self, conn):
"""
Return a connection to the pool.
"""
loop = self.in_use[conn]
del self.in_use[conn]
if loop is not None:
conns, _ = self._ensure_loop(loop)
conns.append(conn)
def conn_error(self, conn):
"""
Handle a connection that produced an error.
"""
conn.close()
del self.in_use[conn]
def reset(self):
"""
Clear all connections from the pool.
"""
self.conn_map = {}
self.in_use = {}
async def close_loop(self, loop):
"""
Close all connections owned by the pool on the given loop.
"""
if loop in self.conn_map:
for conn in self.conn_map[loop]:
conn.close()
await conn.wait_closed()
del self.conn_map[loop]
for k, v in self.in_use.items():
if v is loop:
self.in_use[k] = None
async def close(self):
"""
Close all connections owned by the pool.
"""
conn_map = self.conn_map
in_use = self.in_use
self.reset()
for conns in conn_map.values():
for conn in conns:
conn.close()
await conn.wait_closed()
for conn in in_use:
conn.close()
await conn.wait_closed()
class InvalidConnection(Exception): class InvalidConnection(Exception):
pass pass

View File

@ -83,40 +83,15 @@ DATABASES = {
} }
# Set use_redis to True to activate redis as cache-, asgi- and session backend.
use_redis = False
if use_redis:
# Django Channels
# https://channels.readthedocs.io/en/latest/topics/channel_layers.html#configuration
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("localhost", 6379)],
"capacity": 100000,
},
},
}
# Collection Cache # Collection Cache
REDIS_ADDRESS = "redis://redis:6379/0"
# Can be:
# a Redis URI — "redis://host:6379/0?encoding=utf-8";
# a (host, port) tuple — ('localhost', 6379);
# or a unix domain socket path string — "/path/to/redis.sock".
REDIS_ADDRESS = "redis://127.0.0.1"
# REDIS_READ_ONLY_ADDRESS
AMOUNT_REPLICAS = 1
# Session backend # Session backend
# Redis configuration for django-redis-sessions. # Redis configuration for django-redis-sessions.
# https://github.com/martinrusev/django-redis-sessions # https://github.com/martinrusev/django-redis-sessions
SESSION_ENGINE = 'redis_sessions.session' SESSION_ENGINE = 'redis_sessions.session'
SESSION_REDIS = { SESSION_REDIS = {
'host': '127.0.0.1', 'host': 'redis',
'port': 6379, 'port': 6379,
'db': 0, 'db': 0,
'prefix': 'session', 'prefix': 'session',
@ -144,21 +119,17 @@ ENABLE_ELECTRONIC_VOTING = False
# Internationalization # Internationalization
# https://docs.djangoproject.com/en/1.10/topics/i18n/ # https://docs.djangoproject.com/en/1.10/topics/i18n/
TIME_ZONE = 'Europe/Berlin' TIME_ZONE = 'Europe/Berlin'
# Static files (CSS, JavaScript, Images) # Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.10/howto/static-files/ # https://docs.djangoproject.com/en/1.10/howto/static-files/
STATICFILES_DIRS = [os.path.join(OPENSLIDES_USER_DATA_DIR, 'static')] + STATICFILES_DIRS STATICFILES_DIRS = [os.path.join(OPENSLIDES_USER_DATA_DIR, 'static')] + STATICFILES_DIRS
STATIC_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'collected-static') STATIC_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'collected-static')
# Files # Files
# https://docs.djangoproject.com/en/1.10/topics/files/ # https://docs.djangoproject.com/en/1.10/topics/files/
MEDIA_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'media', '') MEDIA_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'media', '')
@ -169,7 +140,6 @@ MEDIA_ROOT = os.path.join(OPENSLIDES_USER_DATA_DIR, 'media', '')
# Logging # Logging
# see https://docs.djangoproject.com/en/2.2/topics/logging/ # see https://docs.djangoproject.com/en/2.2/topics/logging/
LOGGING = { LOGGING = {
'version': 1, 'version': 1,
'disable_existing_loggers': False, 'disable_existing_loggers': False,

View File

@ -0,0 +1,50 @@
import json
from typing import Any
from django.conf import settings
from typing_extensions import Protocol
from . import logging
from .redis import use_redis
logger = logging.getLogger(__name__)
REDIS_STREAM_MAXLEN = getattr(settings, "REDIS_STREAM_MAXLEN", None)
logger.info(f"Redis stream maxlen {REDIS_STREAM_MAXLEN}")
if use_redis:
from .redis import get_connection
class Stream(Protocol):
def __init__(self) -> None:
...
async def send(self, stream_name: str, payload: Any) -> None:
...
class RedisStream:
async def send(self, stream_name: str, payload: Any) -> None:
fields = {
"content": json.dumps(payload, separators=(",", ":")),
}
async with get_connection() as redis:
await redis.xadd(stream_name, fields, max_len=REDIS_STREAM_MAXLEN)
class NoopStream:
async def send(self, stream_name: str, payload: Any) -> None:
pass
def load_stream() -> Stream:
if use_redis:
return RedisStream()
else:
logger.error("You have to configure redis to let OpenSlides work properly!")
return NoopStream()
stream = load_stream()

Some files were not shown because too many files have changed in this diff Show More