!fzusLjHRJdfvCIRwXh:matrix.org

cylc & isodatetime

7 Members
1 Servers

Load older messages


Timestamp Message
14 Jan 2019
14:30:19@matthewrmshin:matrix.orgMatt Shin Will need to add a version epoch?
14:40:58@matthewrmshin:matrix.orgMatt Shin On the other note. metomi/isodatetime#114 is now ready for review. If this goes in quickly, then we can ask wxtim to skip porting rose date logic in Rose to Python 3 and just use the migrated logic straight from the isodatetime project.
20:08:11@revilo666:matrix.orgHilary Oliver
Will need to add a version epoch?
Wow, PEP 440 is comprehensive; hadn't seen that before.
21:01:12@kinow:matrix.orgBruno P. Kinoshita

Today-I-Learned about version epochs Matt Shin ! Thanks! Looks useful, to make sure we can indicate which is the newer version +1

digression: this is something that is missing (kinda) in Maven. Whenever Java devs search for commons-collections:commons-collections (https://search.maven.org/search?q=commons-collections) the latest version appears to be version 20040616. Of course they changed their schema, and the latest is 3.2.2. They could use the release date, a metadata available in maven, but sometimes projects release older versions with bug fixes... so looks to me like Maven is missing something like this version epoch mechanism from python πŸ˜€

15 Jan 2019
03:39:59@kinow:matrix.orgBruno P. Kinoshita phew, spent the day having fun with travis, kicking it a few times, and preparing a pull request to show a possible way to use setup.py + pytest for the isodatetime new tests. Not sure if perfect, as Travis-CI is still busy building it. Will connect again in a few hours to have a look at the jobs again Matt Shin . But had fun playing with the new isodatetime command!!!! πŸ‘
03:48:29@kinow:matrix.orgBruno P. KinoshitaAnd the way you implemented the command Matt, is exactly what I had in mind for Cylc's command/subcommands (https://github.com/cylc/cylc/issues/2802). A way to encapsulate the logic within Cylc, in classes/functions, with tests. Then re-use these functions internally, and expose via entrypoints in setup.py πŸŽ‰ now we will have a good example for reference! πŸŽ‰
03:59:40@revilo666:matrix.orgHilary Oliver sounds good πŸ‘
05:40:42@kinow:matrix.orgBruno P. KinoshitaHmmm, travis had gotten stuck with python 3.6 and 3.7 before, then I pushed another change to increase the timeout to 50 minutes from 40... then came home, and found that again only 3.6 and 3.7 got stuck, in the same part. An interesting issue!
08:59:53@kinow:matrix.orgBruno P. Kinoshitaimage.png
image.png
09:00:32@kinow:matrix.orgBruno P. Kinoshita Testing tqdm to remove the hack that wwe have to stop travis ci killing out jobs after 10 min with no output... only downside that it adds +1 dependency for tests πŸ€”
09:01:34@kinow:matrix.orgBruno P. KinoshitaAlso tried now to run a part of the timepoint tests concurrently... it takes 0.5 seconds per day... it calculates from 1800 to around 2400 I think... going through every day in each year, and doing some operations... 0.5s per days in years... trying now with 4 threads on travis
09:02:28@kinow:matrix.orgBruno P. Kinoshita 24 mins so far and nothing... the first try failed for python 3.6 and 3.7, but with some other changes, it showed that it was related to the current util.cache... it appears it's not thread safe... so trying functools.lru_cache, which has the same functionality...
09:03:10@kinow:matrix.orgBruno P. KinoshitaBut I can see that I'm digressing too much while reviewing the PR... so probably won't touch that branch today again to sleep on it and think about it again tomorrow with fresh mind πŸ˜₯
09:14:19@matthewrmshin:matrix.orgMatt ShinI believe Travis CI's machines are quite low power, so they are actually not very good with concurrent CPU intensive tasks.
12:38:31@matthewrmshin:matrix.orgMatt Shin

The tqdm module is really cool except that it is not part of the standard library.

We can replace the memoize function here https://github.com/cylc/cylc/blob/master/lib/cylc/cycling/iso8601.py#L60 with functools.lru_cache as well - when we get to porting Cylc to Python 3. (Counting down...)

19:54:30@kinow:matrix.orgBruno P. Kinoshita πŸ‘threading appears to have reduced a bit the time, or at least made the tests pass with 3.6 and 3.7. Will see if I'm able to come up with a good (and simple) pull request for the setup.py. Otherwise will submit one later to avoid blocking you work
19:55:50@kinow:matrix.orgBruno P. Kinoshita And good catch! As soon as 7.8.1 is out I will prepare a pull request for the memoize function. This way we have less risks with this bug fix release.
21:48:21@revilo666:matrix.orgHilary Olivertqdm is cool - let's use it!
21:51:00@kinow:matrix.orgBruno P. Kinoshita

I think it might be useful for cylc.. for commands that take a while to execute and we know how many tasks/jobs/etc we have to wait for (quite sure I have a post-it or card in some github project for that). In the isodatetime case here, it was to a) replace the hack that prevents travis-ci killing jobs due to no output for more than 10 mins, and b) to display how long more the time point tests would take (which we have no idea when looking at the job in travis).

But I agree adding an extra dependency, even for testing, requires some discussion first. The pull request is almost ready without tqdm... but still planning to suggest its use it in isodatetime and/or cylc later :D

21:54:01@revilo666:matrix.orgHilary OliverOnce we're set up with `setup.py` adding a new depedency is really no big deal, right?
21:55:34@kinow:matrix.orgBruno P. Kinoshita Yup, pretty easy. I think UK team has now Python 3. But am not too sure which dependencies they can install ... i.e. are they allowed to just do a pip install tqdm in their environments? If it demands some paperwork... then I think we may hold back adding new dependencies for a while? (though with the new web-gui and Node... hehe, that might be hard)
21:58:07@revilo666:matrix.orgHilary OliverYes, I hope we can all install whatever the hell we need in our own dev environments. I think the UK peeps said they'll be able to do that locally via pip or conda (? is that right?)
21:59:29@kinow:matrix.orgBruno P. Kinoshita (not related: FWIW travis ci servers have 2 cores. In case anybody needs to run anything concurrently there)
22:18:28@matthewrmshin:matrix.orgMatt ShinYes, but we do have a bizarre set of environments at the moment.
22:18:54@kinow:matrix.orgBruno P. KinoshitaI think you guys are using Python 3.4?
22:21:43@matthewrmshin:matrix.orgMatt Shin That's default on RHEL7. We also have the environment from Iris, which is based on 3.6 at the moment, but it lacks essential flexibility in normal mode. We can also rebuild the whole Iris environment from scratch, which will then give us the flexibility.
16 Jan 2019
09:59:14@wxtim:matrix.orgwxtimI'm mostly using 3.6 because that was what was easily available in our scientific software stack. I hope it doesn't cause too many issues later.
18 Jan 2019
16:30:03@matthewrmshin:matrix.orgMatt Shin The 2.0.0 release is almost here. Oliver Sanders is working on updating the README.md with installation instructions and some usage info.
19 Jan 2019
08:06:16@kinow:matrix.orgBruno P. Kinoshita Pull request merged, README.md updated! πŸŽ‰
11:15:46@kinow:matrix.orgBruno P. Kinoshita It looks like we are closing lots of issues (new/old) in isodatetime. So here's a pull request for another one 😁 https://github.com/metomi/isodatetime/pull/118

There are no newer messages yet.


Back to Room List