Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
...
Location: Kirchheim unter Teck
Link to the original pad: https://pad.riseup.net/p/sc4l4bility
| Warning |
|---|
Not proofread by any developer! Read with caution as it may contain mistakes! |
...
RF: Yes, that would be good. Maybe i can start. Our current structure maybe is not the best for the bigger vision and maybe we have to adjust it to make it easier scalable for that we don't have to come up with something basically new in the next two years. That would be the same as we already did it with foodsharing, and that caused a lot of problems. I thought the way we approachd this was suitable for the bigger vision, but apparently it is not.
MS: Maybe before me we missed to talk about how many users to expect, so now we should start addressing this topic, to know how we should build our software.
RF: Also to get a clearer picture of the modular structure, so that we don't have to extract stuff in the future, but to have a nice structure that will function with a variety of different modules.
...
HS: But there is no timeframe. When we talk about 1 year or 2 years, there is a best and a worst case. Maybe 1 million users in one year is something we could work with. 10 million would be the maximum to expect. Those two figures would be best and worst case scenarios.
DW: It would be great to have the foodsharing data available, maybe we could ask Luke, Janina's boyfriend who is a skilled statistician, to have a look and generate a more precise estimate.
...
RF: So what would be your suggestionssuggestion?
PS: We should identify the single points of failure. maybe we need to replace just one piece in the architecture.
...
ML. Yes, we don't design it to work with 20k servers, that wouldn't work with the current system.
PS: I see problems with mysql databases . or that the api can not run in parallel.
(longer explanation about management of requests, insertion of extralayer that eliminates duplicates and decides where exactly to send a request to. it's about queuing requests to not overload the server and/or(?) api. seeking load balance .. key value pair model for nosql vs er model for relational ...see rabbitmq)
ML: So we exchange our database and add a amqp-layer in front. I still don't get why it can't be done later?
...
ML: What is the interface between amqp and api, is it http?
PS: Yes, that or something similar.
ML: But it all is about the entry point of the application, it's not about actually doing something in it, right?
DW: what What in the current architecture would keep us from scaling out, frontend, backend?
...
ML: If we were to use a NoSQL database in the future, than then it wouldn't make sense to use a relational database right now.
...
ML: The way i would build up my data structures would change depending on the database. Since we are talking about scaling and performance here, we should use matching technologies and approaches.
(very technical discussion, in which Pranav shared his idea of how performance could be handled, with millions of users as well. key value pair model for nosql vs er model for relational)
PS: You don't have to worry about the api part at all right now.
...
RF: Okay, i'm really impressed and can basically just trust you to know what you're talking about. So, regarding an outcome of thic this planning meeting...
ML: For me this is still not decided. We agree, that we will need this at some point, but we don't know when. If the outcome was, that we need this in 5 years, i would still stick to the relational databases at the moment. Changing the databases would delay us at least half a year, maybe even a year, because there is not much active development going on right now. But then again, i also don't see how we should release anything in the next 6 months anyways...
ML: It would require a lot of learning, it's a new technology.
...
ML: Well, it wasn't written down, but we addressed the topic, sure. We haven't been very strict on this but the general criteria are clear. we follow the python coding style guidelines. we wanted to do code rewriewsreviews, but since we dont have people doing them, we cantcan't...
ML: I will do more reading on this anyways, and i really think that a framework change is absolutely possible now. basically were strating from zero nowBut then we're basically starting from scratch again...
PS: I think django is a backend and it normally doesn't need to scale out.
...
JA: However, there are definitely a lot of discussions happening in a too private space at the moment. Like Matthias already said, we need to inform the team about what is happening and what kind of decisions are approached right now. If not, there's gonna be massive frustration soon, it actually started already...
RF: Yes we will have a presentation tomorrow in the evening.
...