A conceptional issue in `get_data_since` leads to incomplete
autoupdates. The behaviour was long time in the code, but only with a
lot of autoupdates (high concurrency) and the autoupdate delay I noticed
the bug during testing. I'm sure, that this issue might have caused
incomplete autoupdates (which the user may experience as "lost
autoupdates") in previous productive instances. Instead of quering a
range (from_change_id to to_change_id) one now can only get data from a
change id up to the max change id in the element cache. The max change
id gets now returned by `get_data_since`.
I also added a get_all_data with the capability of returning the
max_change_id at this point of time.
As a usability-"fix" (more like a fix the result of a bug, not the bug
itself) a refresh button for a poll was added, that issues an autoupdate
for the poll and all options.
If autoupdates are too fast, the first one may not be fully executed. Especially when the maxChangeId is not yet updated, the second Autoupdate will trigger a refresh, because for the client it "lay in the future". This can be prevented by synchronizing the autoupdate-handling code with a mutex.
Since channels_redis does not support dedicated read-redis instances, the
autoupdate message may be received before the data was replicated. All workers
read the autoupdate message from the write host, so there is a race between
getting this message and a finished replication. For large payloads, the
replication is slower in the most cases (even more in a distributed setup, where
the master and replica are on different nodes). The easy way is to wait for
replication. But there is one difficulty: The number of replicas has to be
known. There is a new settings-variable "AMOUNT_REPLICAS" which defaults to 1.
It needs to be set correctly! If it is too high, every autoupdate will be
delayed by 1 seconds because of a timeout witing for non-existent replicas. If
it is too low, some autoupdates may be wrong (and not detectable by the client!)
becuase of reading from non-synchronised relicas.
The other possibility is to fork channel_redis and add the feature of a
read-only redis. This ould help, because on a single redis instance all commands
are ordered: First, the data is synced, then the autoupdate message. Attention:
This means, if redis-replicas are scaled up, one must make sure to read from the
same instance. I think this is not possible in the way how dockers overlay
networks work. The only way would be to open one connection and reuse the
connection from channels_redis in OpenSlides. This would mean a heavy
integration of channels_redis (meaning including the source code in our repo).
For the first fix, this one is easy and should work.
- Removed the restricted data cache (it wasn't used since OS 3.0)
- unify functions for restricted and full data: Just one function, which
accteps an optional user_id: If it is None, full data is returned, and
with a user id given, the restricted data
- More atomic access to redis, especially for:
- Check for data-existance in redis and do an auto-ensure-cache.
- Speedup through hashing of scripts and redis' script cache.
- Save schema version into the redis cache and rebuild, if the version
changed
Client changes:
- Simplified the ConstantsService
- Fixed bug, when receiving an autoupdate with all_data=True from the
Server
* Activate restricted_data_cache on inmemory cache
* Use ElementCache in rest-api get requests
* Get requests on the restapi return 404 when the user has no permission
* Added async function for has_perm and in_some_groups
* changed Cachable.get_restricted_data to be an ansync function
* rewrote required_user_system
* changed default implementation of access_permission.check_permission to
check a given permission or check if anonymous is enabled
* delete only keys with prefix
* Make redis_provider atomic with transactions and lua scripts
* improve lock
* generate change_id in redis to make sure it is uniq
* use miliseconds as starttime
* add argument use max_change_id to get_{full|resticted}_data
* geis does not work with channels2 and never will be (it has to be python now)
* pytest
* rewrote cache system
* use username instead of pk for admin user in tests
