I think that something like this helps developers getting oriented in the #damn project. Maybe administrators, too. This graph is not meant for users/mappers.
- Solid links are docker-compose config-related.
- Dashed links are HTTP requests.
- Dotted is just-not-used-now connection.
This is forgotten I/O utilization graph for the last "average" test of 200 mappers and 2 areas.
I did another round of "average" load testing for 200 mappers and 2 areas to maintain the compatibility with https://en.osm.town/web/statuses/106189249266359926 and https://en.osm.town/web/statuses/106189292966532711 (the same number of mappers, the same number of areas.)
The interesting time is at start 11:40 -- spawning of mappers. Next, around 12:13 to 12:18, where mapping is finished (i.e. 0 % to map). Finally, 12:45 -- upkeep script runs while server is returning 404: no square to map. (I checked it with `journalctl -u damn_upkeep.service -f`.)
First round of endurance load testing. Started at 12:00, 0% to map at 15:00, non-recoverable CPU load after 18:00.
My guess is that it's upkeep scripts.
Also, I think that I should speed up waiting of locust average mappers from 30 - 60 seconds to 2 - 5 seconds. I will let the mapping time of 30 - 60 seconds, though.
I think that 200 mappers (30 s mapping, 30 s waiting) with 4 areas is the top wall for load testing. I've tested 300 mappers with 6 areas for about 10 minutes and I don't feel confident about it.
The results of load testing for the server of https://www.damn-project.org/ with 4 areas and 200 mappers (all spawned in the first minute,) and "average" load scenario: 30 - 60 s mapping, 30 - 60 s waiting (https://git.sr.ht/~qeef/damn-server/tree/master/item/tests/mapathon.py) are attached.
Run for 1.5 hour approximately -- after that time there were 0 % to map for all of the areas so the results could become biased.
Server: 1x 2.2 GHz vCPU, 1 GB RAM, 25 GB SSD.
(The images are from the "extreme" scenario of load testing https://git.sr.ht/~qeef/damn-server/tree/master/item/tests/mapathon.py run for few minutes only.)
I had to stop the load test of
with 200 mappers after approximately 20 minutes. (The test performed since 17:00 till 17:30.) The statistics of the areas when stopped were (to map, to review, done:) 9%/66%/25%, respective 37%/47%/16%.
The reason to stop the test was increase in the resource utilization. Unfortunately, I have no database logs this time.
I run the locust.io load testing for
with 100 mappers (all spawned in the first minute,) and "average" load scenario: 30 - 60 s mapping, 30 - 60 s waiting (https://git.sr.ht/~qeef/damn-server/tree/master/item/tests/mapathon.py).
After 3 hours, the statistics of the areas were (to map, to review, done:) 0%/31%/69%, respective 0%/55%/45%.
Overnight test failed (too many 404: no square to map -- that's not an error, but I don't want to test it.) So, I run the average test (each mapper map for 30 - 60 s, then wait for 30 - 60 s before next mapping) again for 3 hours.
For the database queries:
Max duration / Latency Percentile(99)
1 s 190 ms / 32 s 502 ms
From the locust.io testing: 31582 POST requests (the real work) with 99%ile of 420 ms!
2 uvicorn workers, 4 GB RAM, Intel i5-2520M @ 2.5 GHz. Locust.io runs on the same machine.
I've added locust.io config file to the #damn server and performed "extreme" load test. (i.e. I've used "extreme" load times of https://git.sr.ht/~qeef/damn-server/tree/master/item/tests/mapathon.py)
Outcomes are: (1) python server just waits for the database (and it should be like that), (2) about 1 GB RAM is sufficient, (3) need to work on SELECT queries.
Query type: Average / Percentile(99)
INSERT: 2 ms / 634 ms
SELECT: 32 ms / 57 s 834 ms
Next is "average" load test run overnight.
There are four #damn clients now. Lightweight one, proposing the common workflow. The panel that can be opened from the lightweight client. Mappy client for manual square locking and merging. And finally, JOSM plugin.
This is current #damn architecture. Solid links are docker-compose config-related connections. Dashed are HTTP connections. Dotted is not used now.
Dot file with the architecture is here https://git.sr.ht/~qeef/damn-deploy/tree/master/item/architecture.dot
I am working on the #damn project refactor/re-design. The web client is slowly getting done. (Changes on the server side, too, so it's not compatible with the current https://www.damn-project.org/ server. Also, I just see I need to work on statistics yet.)
One big difference will happen. I will allow only common workflow in the _light_ client. That, of course, implies there will be some new _mappy_ client ^^
I tried to test it but not sure how to do it exactly. If you have better idea, I would be happy to know it.
I tried Chromium "Performance" with custom setting for GPRS (50 kbps download, 20 kbps upload, 500 ms latency.)
I tested two requests from the server:
which is comparable to downloaded client.
I could not resist. I put 4000 areas to the #damn server. (HOT TM has 3,093 projects.)
The damn server returns all 4,023 areas in comparable times to TM returning 14 of 3,093.
And I am not talking about "Downloaded client".
The #damn client is static web page. So I added "Download client" link to the bottom. Why?
1. Stable client:
-- too slow as the damn deploy is not optimized.
2. Testing client:
3. Downloaded client:
-- just save it to your desktop and you need to get data from server only.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!