cylc & isodatetime

8 Members
1 Servers

Load older messages

Timestamp Message
15 Jan 2019
09:01:34@kinow:matrix.orgBruno P. KinoshitaAlso tried now to run a part of the timepoint tests concurrently... it takes 0.5 seconds per day... it calculates from 1800 to around 2400 I think... going through every day in each year, and doing some operations... 0.5s per days in years... trying now with 4 threads on travis
09:02:28@kinow:matrix.orgBruno P. Kinoshita 24 mins so far and nothing... the first try failed for python 3.6 and 3.7, but with some other changes, it showed that it was related to the current util.cache... it appears it's not thread safe... so trying functools.lru_cache, which has the same functionality...
09:03:10@kinow:matrix.orgBruno P. KinoshitaBut I can see that I'm digressing too much while reviewing the PR... so probably won't touch that branch today again to sleep on it and think about it again tomorrow with fresh mind πŸ˜₯
09:14:19@matthewrmshin:matrix.orgMatt ShinI believe Travis CI's machines are quite low power, so they are actually not very good with concurrent CPU intensive tasks.
12:38:31@matthewrmshin:matrix.orgMatt Shin

The tqdm module is really cool except that it is not part of the standard library.

We can replace the memoize function here https://github.com/cylc/cylc/blob/master/lib/cylc/cycling/iso8601.py#L60 with functools.lru_cache as well - when we get to porting Cylc to Python 3. (Counting down...)

19:54:30@kinow:matrix.orgBruno P. Kinoshita πŸ‘threading appears to have reduced a bit the time, or at least made the tests pass with 3.6 and 3.7. Will see if I'm able to come up with a good (and simple) pull request for the setup.py. Otherwise will submit one later to avoid blocking you work
19:55:50@kinow:matrix.orgBruno P. Kinoshita And good catch! As soon as 7.8.1 is out I will prepare a pull request for the memoize function. This way we have less risks with this bug fix release.
21:48:21@revilo666:matrix.orgHilary Olivertqdm is cool - let's use it!
21:51:00@kinow:matrix.orgBruno P. Kinoshita

I think it might be useful for cylc.. for commands that take a while to execute and we know how many tasks/jobs/etc we have to wait for (quite sure I have a post-it or card in some github project for that). In the isodatetime case here, it was to a) replace the hack that prevents travis-ci killing jobs due to no output for more than 10 mins, and b) to display how long more the time point tests would take (which we have no idea when looking at the job in travis).

But I agree adding an extra dependency, even for testing, requires some discussion first. The pull request is almost ready without tqdm... but still planning to suggest its use it in isodatetime and/or cylc later :D

21:54:01@revilo666:matrix.orgHilary OliverOnce we're set up with `setup.py` adding a new depedency is really no big deal, right?
21:55:34@kinow:matrix.orgBruno P. Kinoshita Yup, pretty easy. I think UK team has now Python 3. But am not too sure which dependencies they can install ... i.e. are they allowed to just do a pip install tqdm in their environments? If it demands some paperwork... then I think we may hold back adding new dependencies for a while? (though with the new web-gui and Node... hehe, that might be hard)
21:58:07@revilo666:matrix.orgHilary OliverYes, I hope we can all install whatever the hell we need in our own dev environments. I think the UK peeps said they'll be able to do that locally via pip or conda (? is that right?)
21:59:29@kinow:matrix.orgBruno P. Kinoshita (not related: FWIW travis ci servers have 2 cores. In case anybody needs to run anything concurrently there)
22:18:28@matthewrmshin:matrix.orgMatt ShinYes, but we do have a bizarre set of environments at the moment.
22:18:54@kinow:matrix.orgBruno P. KinoshitaI think you guys are using Python 3.4?
22:21:43@matthewrmshin:matrix.orgMatt Shin That's default on RHEL7. We also have the environment from Iris, which is based on 3.6 at the moment, but it lacks essential flexibility in normal mode. We can also rebuild the whole Iris environment from scratch, which will then give us the flexibility.
16 Jan 2019
09:59:14@wxtim:matrix.orgwxtimI'm mostly using 3.6 because that was what was easily available in our scientific software stack. I hope it doesn't cause too many issues later.
18 Jan 2019
16:30:03@matthewrmshin:matrix.orgMatt Shin The 2.0.0 release is almost here. Oliver Sanders is working on updating the README.md with installation instructions and some usage info.
19 Jan 2019
08:06:16@kinow:matrix.orgBruno P. Kinoshita Pull request merged, README.md updated! πŸŽ‰
11:15:46@kinow:matrix.orgBruno P. Kinoshita It looks like we are closing lots of issues (new/old) in isodatetime. So here's a pull request for another one 😁 https://github.com/metomi/isodatetime/pull/118
22 Jan 2019
22:50:37@matthewrmshin:matrix.orgMatt ShinWhen https://github.com/metomi/isodatetime/pull/120 is merged, I'll tag the release and prepare an upload for PyPI.
23 Jan 2019
03:44:18@kinow:matrix.orgBruno P. KinoshitaπŸ‘ just approved #120
03:44:29@kinow:matrix.orgBruno P. KinoshitaπŸ•Ί
12:54:39@matthewrmshin:matrix.orgMatt ShinRelease 2.0.0 tagged and pushed to Github. I'll update PyPI tonight.
15:03:26@matthewrmshin:matrix.orgMatt Shinhttps://pypi.org/project/isodatetime/ - new version up.
18:42:23@kinow:matrix.orgBruno P. KinoshitaπŸ‘ πŸ‘ πŸ‘
19:46:12@sadielbartholomew:matrix.orgSadie BartholomewGreat job everyone πŸŽˆπŸŽ‰βœ¨
21:33:03@revilo666:matrix.orgHilary Oliver πŸ’₯
18 Mar 2019
10:48:47@kinow:matrix.orgBruno P. Kinoshita invited @wxtim2:matrix.orgwxtim2.
10:48:47@wxtim2:matrix.orgwxtim2 joined the room.

There are no newer messages yet.

Back to Room List