The Geek Lab - Public Room Timeline

	The Geek Lab	939 Members
	A place to discuss Linux and open source related technologies. Please keep conversations family friendly	112 Servers

Load older messages

Sender	Message	Time
26 Jan 2022
CubicleNate	No problem!	15:26:22
Kpovoc \| LD	We’re getting an upgrade!	15:27:04
prozak	Thank you for providing means to this community to exist Noah (Kernellinux) you rock	15:27:49
m. x. u.	Thank you Noah (Kernellinux) for making this all available in the first place! You are awesome!	15:28:31
Kpovoc \| LD	In reply to @prozak:prozak.org Thank you for providing means to this community to exist Noah (Kernellinux) you rock 100% this. Noah (Kernellinux) Providing this as a free service is beyond generous (some might say ridiculous). Thank you for all you do.	15:29:32
RyuKurisu	In reply to @kernellinux:linuxdelta.com @room Matrix Server Update Back in 2020 we trialed Matrix to be used at SELF. It was a bit touch and go at that time (mostly due to our inexperience of managing Synapse). We opted to use an alternative but I was disappointed that we had this new cool new open source tool that was now not going to be used, and half-way through SELF I posted a message that said something to the effect "We're spinning this up and going to play with it - it might crash and burn but it will be a fun experiment" That "experiment" turned into a fantastic way to connect and communicate with people from all over the world in a secure and de-centralized way. That hasn't been without problems, but the juice has very much been worth the squeeze. What Happened? Many of you may have noticed that our Matrix server has had some issues recently. This is due to storage. This server has grown faster than I would have ever anticipated. This server is what is used to provide my personal bridges. This server is what is used for many of our bridges at Altispeed, and of course runs Ask Noah Show. I consider it to be absolutely essential and I will continue to pump whatever resources are necessary to expand and support our growing community and support this platform. What Happens Next? We are in the process of setting up a new home for our Matrix server that will allow us to grow without constraints. We anticipate most of the work being completed by end of day today. We have about 60G left to copy, at some point it will be necessary to power off this instance for a brief time, make sure our data is up to date, then power up our new server and make it live. I apologize sincerely for those of you who have experienced outages and the frustration that accompanies it. We will make it better! Thank you for being in this room, and thank you for being a part of this community! 100% uptime is never really possible; it is better to have planned outages!	15:46:40
prozak	In reply to @kernellinux:linuxdelta.com @room Matrix Server Update Back in 2020 we trialed Matrix to be used at SELF. It was a bit touch and go at that time (mostly due to our inexperience of managing Synapse). We opted to use an alternative but I was disappointed that we had this new cool new open source tool that was now not going to be used, and half-way through SELF I posted a message that said something to the effect "We're spinning this up and going to play with it - it might crash and burn but it will be a fun experiment" That "experiment" turned into a fantastic way to connect and communicate with people from all over the world in a secure and de-centralized way. That hasn't been without problems, but the juice has very much been worth the squeeze. What Happened? Many of you may have noticed that our Matrix server has had some issues recently. This is due to storage. This server has grown faster than I would have ever anticipated. This server is what is used to provide my personal bridges. This server is what is used for many of our bridges at Altispeed, and of course runs Ask Noah Show. I consider it to be absolutely essential and I will continue to pump whatever resources are necessary to expand and support our growing community and support this platform. What Happens Next? We are in the process of setting up a new home for our Matrix server that will allow us to grow without constraints. We anticipate most of the work being completed by end of day today. We have about 60G left to copy, at some point it will be necessary to power off this instance for a brief time, make sure our data is up to date, then power up our new server and make it live. I apologize sincerely for those of you who have experienced outages and the frustration that accompanies it. We will make it better! Thank you for being in this room, and thank you for being a part of this community! Noah (Kernellinux): now something i am pondering about server migration is about the data copy backup, my understanding is that by design the federation portion of synapse may take care of the media storage, as in, once bridged the servers a copy is made in the environment, can/was this something that you guys are leveraging as part of the migration process? As an owner of a tiny personal server i would be delighted to hear the journey recap in the show	17:46:11
Sleuth	I can't speak to exactly how they are doing it. But from what Noah has told me they are coping the files from the old server to the new one. The only reason you would use federation is if he were to change domains. And that would be a very very long and tedious process for both him and all the users. Everyone would essentially have to make a new account and join it to all the rooms they are in.	18:09:11
Sleuth	Federation is for between server communication not to move a server. He is just moving hardware not changing how the server works.	18:09:44
Sleuth	* Federation is for real time server to server communication not to move a server. He is just moving hardware not changing how the server works.	18:10:15
midek	Hey Noah (Kernellinux) is it the DB that grew so quickly for you? or perhaps media storage? you can safely purge the media storage from remote servers that is older than couple months. If its the whole DB thats growing too fast, have you seen this? https://levans.fr/shrink-synapse-database.html I have been running a medium synapse server for about 5 years now, and shrinking like that helped me a lot over the years, my postgres DB is around 400GB currently. Feel free to @ me or DM if you need some additional help maintaining synapse server, i could try to get you in touch with other synapse admins and perhaps they can also help.	19:12:33
Jivan Pal 🇬🇧	my postgres DB is around 400GB currently. Oof.	19:13:34
midek	In reply to @jivan:opaline.uk > my postgres DB is around 400GB currently. *Oof.* well the db is on 1tb raid1 ssd mirror so far away from panic for now :p	19:14:10
Jivan Pal 🇬🇧	Haha, yeah, I hope you've got the space for it, it's just that 400GB is more data than I own in total, lol	19:16:18
Jivan Pal 🇬🇧	My Synapse instance which this account I'm using is on is "only" 23GiB in the database, 4.3GiB other files incl. media	19:17:20
Jivan Pal 🇬🇧	It's only joined to like 15 rooms though, mainly on Matrix.org, I set it up about 6 months ago	19:18:49
midek	In reply to @jivan:opaline.uk Haha, yeah, I hope you've got the space for it, it's just that 400GB is more data than I own in total, lol that 400GB is just the postgres DB, i keep media store separate on spinning rust, and they are over 1TB i will probably purge some remote content again to keep media storage under control, because it does grow like crazy	19:23:19
Sleuth	midek: the current issue is that the Linux Delta synapse is on DO. So storage is a bit pricey. Noah just racked several servers for altispeed so it makes sense to move the server to a place that he controls and has more storage. I'll tell Noah that you are willing to help and might have some contacts with other devs.	20:12:49
midek	In reply to @sloth56:chat.mountainview.theworkpc.com midek: the current issue is that the Linux Delta synapse is on DO. So storage is a bit pricey. Noah just racked several servers for altispeed so it makes sense to move the server to a place that he controls and has more storage. I'll tell Noah that you are willing to help and might have some contacts with other devs. ahh, yeah i can imagine hosting synapse could get very expensive very fast on a VPS	20:14:24
Sleuth	Yeah, so it's upgrade to the next tier which almost doubles the price per month or figure out block storage. When altispeed was purchasing all those server's Noah thought to have enough extra space to move the synapse to it eventually. So this isn't entirely out of left field just needed done sooner than expected.	20:16:56
Sleuth	The move to the new infra has been a slow and calculated one. As it should be. They are being very careful while they are still testing it all out. It's been rock solid so it was unanimous at the production meeting to move it.	20:19:15
midek	So everything is under control, good to hear.	20:19:56
tiny	Not that it matters since they are moving but did they setup object storage for media?	20:47:49
Warped	I'm mostly in the rooms from this community and a couple others that are probably common interest. I have stayed away from big busy rooms that would tax a server. I still have an IRC client with about 30 channels that logs, so that reduces the Matrix rooms I might join that just replicate IRC channels. Eventually I'll host my own Matrix instance. Thanks Noah for all the work you've been doing that we all benefit from.	20:58:16
Sleuth	In reply to @tiny6996:matrix.org Not that it matters since they are moving but did they setup object storage for media? Not sure, but I don't think so.	22:26:08
	Niels joined the room.	22:48:31
27 Jan 2022
	Tom ,🇨🇦🏳️‍🌈💙 joined the room.	00:20:10
	Allegra Padula joined the room.	01:51:05
	Tom ,🇨🇦🏳️‍🌈💙 set a profile picture.	02:09:48
Ninja	I had a fast VPS for my Synapse for a while, after hosting it myself on my old slow hypervisor. It allowed me to join many more rooms and have a fast, responsive system. I trimmed down the number of rooms I'm in, and eventually moved off of the VPS. Migrating from one server to another was a breeze. Initial data syncs could be done while the server is live, but a final sync needs to be done while postgresql and synapse are stopped. (There's another way to do it, but this is the easiest and simplest.) I'm considering moving back to a hosted server. Maybe GCE will be my new home, maybe Linode, maybe Vultr. I haven't decided, yet. But, it's just me on my own server, so I have complete control over retention and resource management. For those who can run their own server, it's a much better experience.	02:10:52