| 25 Nov 2020 |
 @gitter_lexiconcode:matrix.org | Greetings @aschneiderman Well there are a number of different techniques. You could use image recognition with Sikulix or you could use Selenium. Sikulix Is a bit fragile and doesn't integrate very well to grammars but it is functional. Selenium Is definitely doable and I've seen some grammars use it however requires a plug-in and some set up but it is also very doable. Possibly harder than the other 2 but I think a lot more generalizable is using accessibility APIs like UI automation from Windows to grab anything from most programs. pywinauto it is probably the best Python is the one best implementation. Accessibility Insights For Windows Allows you to inspect live applications to assist in leveraging UI automation. There are a few things missing for text navigation pywinauto but hopefully that can be resolved in future. | 02:45:33 |
| @gitter_lexiconcode:matrix.org | Here's an example from pywinauto save_from_firefox.py Automating the UI. I give you a point of reference you won't really need to emulate keys as you can do that to dragonfly. | 02:52:56 |
| @gitter_lexiconcode:matrix.org | | 02:53:00 |
| @gitter_lexiconcode:matrix.org | (edited) [](https://files.gitter.im/55dba9af0fc9f982beae7a63/TIUG/image.png) => Accessibility Insights For Windows to show the address bar
[](https://files.gitter.im/55dba9af0fc9f982beae7a63/TIUG/image.png) | 02:53:40 |
| @gitter_lexiconcode:matrix.org | Here's an example from pywinauto save_from_firefox.py Automating the UI. A point of reference as you won't really need to emulate keys as you can do that to dragonfly. | 03:15:16 |
 @gitter_tteig:matrix.org | tw | 03:19:35 |
 @gitter_aschneiderman:matrix.org | Thanks @LexiconCode! I’m using a Windows 10 box, so I can only use Jira via the web, so pywinauto and Accessibility Insights for Windows would give me a lot less semantic info than a tool like Selenium that can read elements of the DOM, right? And Sikulix would be helpful if I was clidking on static images, but it wouldnt work well with dynamic lists pf user stories. Or am I missing what you’re saying? | 06:03:45 |
| @gitter_aschneiderman:matrix.org | (edited) ... was clidking on static images, but it wouldnt work well with dynamic lists pf user ... => ... was clicking on static images, but it wouldn't work well with dynamic lists of user ... | 06:05:07 |
| @gitter_aschneiderman:matrix.org | Has anyone used Selenium with Caster or seen grammars that do? I did o | 06:31:11 |
| @gitter_aschneiderman:matrix.org | (edited) ... did o => ... did a bit of Googling and didn’t find exs, so if you know of one, can you post a link? I’m fluent in Python but haven’t written anything either with Caster or Selenium before, so it’d be great to reduce the time needed to figure out connecting the two. Thanks! | 06:33:28 |
| @gitter_lexiconcode:matrix.org | I'm pretty sure you can get access to anything on the Dom as it's used by screen readers in other projects. However that's more of an educated guess than actual experience. For your second request here you go https://github.com/wolfmanstout/dragonfly-commands/blob/master/_webdriver_utils.py | 07:14:28 |
| @gitter_lexiconcode:matrix.org | (edited) ... request here ... => ... request implemented in dragonfly here ... | 07:16:39 |
| @gitter_lexiconcode:matrix.org | That should give you enough to implement the driver and so forth. You might have to poke around a little bit more to find the grammar that utilizes the web driver. | 07:18:01 |
| @gitter_lexiconcode:matrix.org | Let me know how it goes as I'd love to see something like this pulled upstream into the project. | 07:22:03 |
| @gitter_aschneiderman:matrix.org | Thanks so much, @LexiconCode! | 11:34:09 |
| 26 Nov 2020 |
| @gitter_aschneiderman:matrix.org | I’m starting to experiment with craeting | 01:40:16 |
| @gitter_aschneiderman:matrix.org | (edited) I’m starting to experiment with craeting => Is there a page somewhere that has a list of links to different users’ custom grammars? I couldn’t see one in the docs. I’ve been using Vocola for years and am starting to switch to Caster, so it’d be useful to get a sense of how different users customize Caster. Thanks! | 01:43:45 |
| @gitter_lexiconcode:matrix.org | @aschneiderman You can reference the people on this channel and the following list dictation-toolbox.org. Note you're not limited to just Caster grammars/dragonfly is pretty much compatible inside the mappings as long as the grammars are separate for each file and you have the caster loading mechanism of get rule. | 03:22:44 |
| @gitter_aschneiderman:matrix.org | @LexiconCode Thanks! | 12:26:20 |
 @gitter_patricktouchette:matrix.org | @aschneiderman an idea that comes to mind, since I am working on something similar, is to create a chrome (firefox, etc.) extension which contains your dom manipulation javascript. Bind those functions to keyboard shortcuts, add a rule to caster and then bam, you can invoke those functions with your voice. Eg. "complete task" could find the dom element and click it for you or something along those lines. | 17:21:24 |
| 28 Nov 2020 |
| @gitter_aschneiderman:matrix.org | @patricktouchette Nice idea! Do you have a GH repo where you’ve been playing? | 00:44:21 |
| @gitter_aschneiderman:matrix.org | @patricktouchette The more I think about it, the more this seems like the way to go. The syntax looks pretty straightforward, and not onky can you add keyboard shortcuts, you can easily create popup dialog boxes, add links, and maintain state between pages — eg, between a page of a list of user stories or tasks and the page that lets you edit an individual item. You can also create a text box and process what you voice in it, so for example you could say “mark 2, 5, 8” that would put “2,5,8” in the tet | 12:28:23 |
| @gitter_aschneiderman:matrix.org | (edited) ... item. You can also create a text box and process what you voice in it, so for example you could say “mark 2, 5, 8” that would put “2,5,8” in the tet => ... item. I have a little time in the next week to play with it; I’ll post what I learn | 12:30:16 |
| @gitter_aschneiderman:matrix.org | (edited) ... not onky can ... => ... not only can ... | 12:30:39 |
 @gitter_itsrainingschmen:matrix.org | Is there a reason why nobody has built any caster commands for wordprocessing programs? Seems like a fairly simple thing to do to get Microsoft Word working with it. If there is no good reason not to do it, I'll likely make that a Christmas break project of mine. | 17:52:22 |
 @gitter_alexboche:matrix.org | Dragon has some decent select and say commands which work in Word. Main drawback is the excessive required wait time between utterances. | 19:42:25 |
| @gitter_alexboche:matrix.org | Personally, I would encourage people interested in investing in improving Caster to consider working on improved eyetracking capabilities. I believe eyetracking needs to be a part of any handsfree users toolkit and is likely to be more effective and generalizable than other approaches | 19:45:23 |
| @gitter_alexboche:matrix.org | (edited) ... other approaches => ... other approaches-- there is a good reason that mice are popular | 19:45:43 |
| @gitter_alexboche:matrix.org | (edited) ... and is likely to be ... => ... and will be ... | 19:46:10 |
| @gitter_alexboche:matrix.org | (edited) ... other approaches-- there is a good reason that mice are popular => ... other approaches | 19:46:47 |
