Present: Raphael Fellmer, Pranav Salunke, Matthias Larisch, Holger Schulz, Martin Schott, Douglas Webb, Janina Abels

Notary: Janina Abels

Location: Kirchheim unter Teck

Link to the original pad: https://pad.riseup.net/p/sc4l4bility

Not proofread by any developer! Read with caution as it may contain mistakes!

[Meeting starts: 14:45]

Expectations

RF: To align the intentions, i'd say we're here to reevaluate the current architechture of the product and to clarify the vision from a technical point of view.

JA: This is a planning meeting then, where there should be outcomes in the end?

RF: Yes, that would be good. Maybe i can start. Our current structure maybe is not the best for the bigger vision and maybe we have to adjust it to make it easier scalable for that we don't have to come up with something basically new in the next two years. That would be the same as we already did it with foodsharing, and that caused a lot of problems. I thought the way we approachd this was suitable for the bigger vision, but apparently it is not.

MS: Maybe before we missed to talk about how many users to expect, so now we should start addressing this topic, to know how we should build our software.

RF: Also to get a clearer picture of the modular structure, so that we don't have to extract stuff in the future, but to have a nice structure that will function with a variety of different modules.

ML: If this is still about the intentions of this meeting i'd say that i agree. That's also what i expect.

Minutes

MS: So let's start with the requirements we want to be able to meet. When we talk for example about users, it's hard to guess what to expect. I have a rough estimate based on the foodsharing users.

RF: Also, foodsharing is just one module, there will be others, too. In the vision, there is a lot more than just foodsharing.

HS: The huge vision is to connect all people in yunity.

ML: Yes, the huge vision would be maybe 3 billion people, so maybe half of the people who have internet.

HS: But there is no timeframe. When we talk about 1 year or 2 years, there is a best and a worst case. Maybe 1 million users in one year is something we could work with. 10 million would be the maximum to expect.

DW: It would be great to have the foodsharing data available, maybe we could ask Luke, Janina's boyfriend who is a skilled statistician, to have a look and generate a more precise estimate.

JA: I already wrote an email to him yesterday and he said he'd have a look. Don't know how much time he could spend on this at the moment, but he didn't say no directly, so...

HS: I think it's the wrong way to think in the size of foodsharing, yunity will much bigger anyways. And it also depends on our belief, if we think it's possible or not.

ML: I still think a million users is something we could handle in our current approach. We are not building a crap software that is not made for being used. But if we think about much higehr numbers, we may really have to rethink our design. But again, we didn't think so far ahead, we don't have a clear architecture at the moment. And we don't see it necessary now to split everything up already.

PS: Are you sure that the existing system can scale that much?

ML: There is no existing system.

PS: Just to scale the database you need to add another management layer. You'd have to build a new management layer on top of the relational database layer and that's really really complex. I spoke to someone from wikimedia and they only did it because at the time there was no other possibility.

ML: So you suggest that it's impossible to run a multimillion users project with relational databases?

PS: Yes, because we'd need to invest in mainframes and things like that and that would be very very expensive.

RF: So what would be your suggestion?

PS: We should identify the single points of failure. maybe we need to replace just one piece in the architecture.

ML: We don't know our points of failure yet, so we don't have a detailed image of that. We cannot replace anything because we don't have anything yet.

PS. We could learn from the mistakes of others. I know for sure that trying to scale this to 20k servers wouldn't work.

ML. Yes, we don't design it to work with 20k servers, that wouldn't work with the current system.

PS: I see problems with mysql databases or that the api can not run in parallel.

(longer explanation about management of requests, insertion of extralayer that eliminates duplicates and decides where exactly to send a request to. it's about queuing requests to not overload the server and/or(?) api. seeking load balance ...see rabbitmq)

ML: So we exchange our database and add a amqp-layer in front. I still don't get why it can't be done later?

PS: It will simply be more complex later.

ML: What is the interface between amqp and api, is it http?

PS: Yes, that or something similar.

ML: But it all is about the entry point of the application, it's not about actually doing something in it, right?

DW: What in the current architecture would keep us from scaling out, frontend, backend?

PS: The frontend basically has no impact on that.

ML: In the backend it's mainly the database.

HS: Is there a way to prepare what you, Pranav, skeched, to implement it later on? I think we are on the same side, regarding that this should be our end solution, because it will be a necessity in the future. Whatever we do right now it should make this possible. It doesn't have to be programmed right now, but we shouldn't programm anything that would prevent this.

ML: If we were to use a NoSQL database in the future, then it wouldn't make sense to use a relational database right now.

HS: Yes.

ML: The way i would build up my data structures would change depending on the database. Since we are talking about scaling and performance here, we should use matching technologies and approaches.

(very technical discussion, in which Pranav shared his idea of how performance could be handled, with millions of users as well. key value pair model for nosql vs er model for relational)

PS: You don't have to worry about the api part at all right now.

ML: Thank you, i think i understood it now.

RF: Okay, i'm really impressed and can basically just trust you to know what you're talking about. So, regarding an outcome of this planning meeting...

ML: For me this is still not decided. We agree, that we will need this at some point, but we don't know when. If the outcome was, that we need this in 5 years, i would still stick to the relational databases at the moment. Changing the databases would delay us at least half a year, maybe even a year, because there is not much active development going on right now. But then again, i also don't see how we should release anything in the next 6 months anyways...

ML: It would require a lot of learning, it's a new technology.

HS: We need some people who already know about this. And then they can teach the others, you'd not only be learning for yourself.

PS: It's also not easy to learn these things, they're quite hidden, even in the technical world.

ML: I'm already learning and reading and am up for wrapping my head around this. I think i'll even have fun with it, since i like this kind of architecture.

RF: I want to do everything possible to prepare the field and use every possible means to enable people who are 100% commited to 100% focus on the project, so that we can release something in one year the latest.

ML: It's also about actually running the software...

PS: That is a dev ops questions.

ML: No i mean, we will need physical technical infrastructure.

ML: It's not about the number of users it's about the number of simultaneous requests, which could be estimated by users being online at the same time.

ML: foodsharing has 20k active users, 120 users peak, so let's say 0,8-1% peak.

RF: What are the concrete downsides to saying we use a smart, prepared architecture today?

HS: If we don't maintain high-level quality code we won't be able to get new devs and we'll get stuck. but maybe it makes sense to switch the database.

PS: Yes, from my perspective that's the one thing required.

HS: Do we already have development guidelines?

ML: Well, it wasn't written down, but we addressed the topic, sure. We haven't been very strict on this but the general criteria are clear. we follow the python coding style guidelines. we wanted to do code reviews, but since we dont have people doing them, we can't...

ML: I will do more reading on this anyways, and i really think that a framework change is absolutely possible now. But then we're basically starting from scratch again...

PS: I think django is a backend and it normally doesn't need to scale out.

ML: It's also about nosql support and i think django lacks a bit in that.

JA: So... concerning tangible outcomes of this meeting, are there any?

RF: Yes, i think we decided to expect fast growth and need to design the yunity platform accordingly.

ML: I don't see this as a decision, because i don't think we can make a decision like this in this small group.

RF: Of course we can make this decision.

ML: I see this as a syscon definitely, because this impacts the whole project, and i don't think that everybody expects figures like 10 million users in a year. I can't make decisions for everyone else. I just can make the decision for myself, that i will dive into research mode again for the next weeks, but even if it's 'just' about development, i don't want to decide for a group of people.

RF: But are we all visionaries? i think many people are not able to think big at the moment, and we need to believe that we can make it or of course we won't. in every project there are just some people who have the bigger vision and then there are people who are attracted by it. If we were to listen to every restistance nothing would happen.

HS: I think this decision has already been made.

ML: Yeah well then we need to have it written down so that we can work with it at least. Also, we really need to communicate it more clearly to everyone involved.

HS: There seems to be a gap between the visions we have here. Apart from the technical side, do you share this vision of changing the world with yunity? If we were to have a million users in 6 months, how would you feel about that?

ML: I... It's a difficult question i think i don't want to answer right now.

JA: However, there are definitely a lot of discussions happening in a too private space at the moment. Like Matthias already said, we need to inform the team about what is happening and what kind of decisions are approached right now. If not, there's gonna be massive frustration soon, it actually started already...

RF: Yes we will have a presentation tomorrow in the evening.

[Meeting ends: 16:45]

Foodsaving Worldwide

2016-05-25 Scope and scalability of yunity

Expectations

Minutes