Sorry that it is hard
And I am sorry, so here are...
My advice: start with some experimental traces and tracks and see if they make sense to you, and if anything written in this issue helps.
If you come up with an interesting (twisted) new use case, or if you can't make sense of your results at all, please come back here and somebody will try to help. This is how we improve the ontology, vocabulary, documentation, and code being developed. Some of us actually like trying to figure the twisted ones out...
Also, note, so far this is a proposal, not yet in the vocabulary. So, it's a good time to suggest improvements or "bad idea, forget it", etc.
@sbocconi (who might not be pingable here) is thinking about ULIDs which are supposed to be sortable and include a Lamport timestamp, which might be a better solution. But those need to be based on processes and events, so we might come back to some of the same causal breadcrumbs...
I edited my comment above to strike out breaking...
Well in terms of the graphql API, versioning isn't necessary especially for additions (removed fields can be marked as deprecated in the schema) https://graphql.org/learn/best-practices/#versioning
Thanks @mayel this is helpful!
Yeah, this isn't actually a breaking change. It's an addition, and I don't think anyone has written track and trace yet in this current round of projects.
[edit] There is an effect if people have been saving lots of economic events, since the solution writes extra information there as they are created. I don't think that is the case yet either.
Well in terms of the graphql API, versioning isn't necessary especially for additions (removed fields can be marked as deprecated in the schema) https://graphql.org/learn/best-practices/#versioning
So the VF ontology does need versioning now, right?
Great spinoff issue! This is a good place to start some discussion about it and then, with your permission, I'd like to start another issue with that as the whole focus.
Question back to you and anybody else who wants to chime in:
How many VF implementations in actual use now? And can they all implement this change? If so, does the vocabulary need to be versioned?
Several of the implementers are in that list of mentions in my comment upthread. Maybe they could help us think about it?
I am way to unknowledgeable for having an opinion on this, but I heard ... BREAKING CHANGE!! :D
So the VF ontology does need versioning now, right?
@srfsh @pospi @mayel @adaburrows @vortex @Connoropolous @hoijui @ivan
Please look at this issue and its suggested solution and pseudocode critically. @lynnfoster and I have been desk-checking this weekend and think it is ok, but it will require a [edit] breaking change [might not be breaking - see @lynnfoster commments below] to any valueflows implementation code that creates EconomicEvents, to add and populate those two new fields, EconomicResource.previousEvent and EconomicEvent.previousEvent.
Those are necessary for providing breadcrumbs for causally ordering tracking and tracing of resource flows.
Now is the time to yell if you see something wrong.
Thank you all very much.
Lynn posted her pseudocode before I got this more-detailed version of the suggested solution posted, but here is...
Add two new fields called previousEvent, one on EconomicResource and one on EconomicEvent. The previousEvent fields reference an EconomicEvent.
When a new event is created with a resourceInventoriedAs, the new event takes the resource's "previousEvent" value and puts it into the new event's "previousEvent" field, and puts the new event into the resource's "previousEvent" field.
Those previousEvent references provide breadcrumbs for tracking and tracing through ambiguous situations. In the table below, you can see an EventID jump from the Event column in one row over to the left, to the Resource.previousEvent column on the next row, and then be reproduced over to the right in the Event.previousEvent column.
Resource | Resource.previousEvent | Event | Event.previousEvent |
---|---|---|---|
Gown1 | Produce1 | ||
Gown1 | Produce1 | Transfer1 | Produce1 |
Gown1 | Transfer1 | Accept1 | Transfer1 |
Gown1 | Accept1 | Modify1 | Accept1 |
Gown1 | Modify1 | Accept2 | Modify1 |
Gown1 | Accept2 | Modify2 | Accept2 |
Gown1 | Modify2 | Accept3 | Modify2 |
Gown1 | Accept3 | Modify3 | Accept3 |
Logic
Except for the repetetive process situation described above, the logic described in https://www.valueflo.ws/algorithms/track/#track-and-trace-logic should work for accurate causal ordering.
If you are tracing a resource, and you find more than one of the EconomicEvents affecting it that are outputs from Processes, or transfer/move EconomicEvents with the resource defined as the toResourceInventoriedAs, use the resource's previousEvent reference as the "before" event.
If you are tracking a resource, and you find more than one EconomicEvents affecting it that are inputs to Processes with the resource defined as the resourceInventoriedAs, if one of the events has no previousEvent, use that one.
If more than one of the events is an input to a process with the same ProcessSpecification, you are in a loop in the resource flow. In that case, usually that process will have an as many output events as input events for the same resource, and each output event should reference one of the input events. You may need to track each of the output events until the sequence of events loops back to another process with the same ProcessSpecification, and see which of the other input events the sequence loops back to. If the sequence does not loop back to a process with the same ProcessSpecification, the event that started this sequence was the causally-ordered last of the multiple events that you started this loop with. If the sequence does loop back to a process with the same ProcessSpecification, the input event it loops back to is causally ordered after the event that started the sequence, and the event that you started with is causally ordered before the event that looped back. (I know this description is loopy but so is the situation.) You may need to take each of the multiple events you found at the beginning of these loops and track it through until it loops back to a process with the same ProcessSpecification. When you are finished, each of the multiple events should be causally sequenced relative to the other input events for the same resource.
If you are lucky, the loops are short, and the resource being tracked goes through very few events before returning to the repetitive process (in the case of the medical gowns, being worn and then washed many times in their life cycle).
If not,if all of the events are inputs to different Processes (that is, with different ProcessSpecifications)? That's not the repetitive process situation, and most likely, the events are not causally connected. If they are not causally connected to each other, their sequence does not matter, so start with the smallest event id: that will provide more consistent results from multiple tracking runs. What if you end up in a loop? Then those events are causally connected, so one of them happened before the other. See previous paragraphs for how to causally order. Sorry that it is hard. So is life in loopy resource flows.
Here is my 3rd draft of the pseudocode, just for trace. I've included logic to handle Bob's solution to this issue, just for trace. This version is not even yet desk checked against data on paper, yet alone tested. If it gets coded by @srfsh before we do that desk checking, then we'll be able to let the computer do the work. If not, @bhaugen and I can do some detailed desk checking with the data from @srfsh , which matches the first diagram, and hopefully clear out some of the bugs. :) At some point, we'll want to do the same kind of thing with the second diagram.
The solution for just trace from @bhaugen :
Add two new fields called previousEvent, one on EconomicResource and one on EconomicEvent. The previousEvent fields reference an EconomicEvent.
When a new event is created with a resourceInventoriedAs, the new event takes the resource's "previousEvent" value and puts it into the new event's "previousEvent" field, and puts the new event into the resource's "previousEvent" field.
Those previousEvent references provide breadcrumbs for tracking and tracing through ambiguous situations.
If you are tracing a resource, and you find more than one of the EconomicEvents affecting it that are outputs from Processes, or transfer/move EconomicEvents with the resource defined as the toResourceInventoriedAs, use the resource's previousEvent reference as the "before" event.
The pseudocode:
Note the "starting item" can be a resource or event. The event is used when wanting to start the trace earlier than the present time, for example to trace only what went into initial creation of the resource. If it is an event related to a process, it is assumed to be an output event. If you want to start the trace at the present time, use the resource to be traced.
trace (parameter: starting item)
initialize "flows", "visited", "contained", "modified", "delivered"
(list or similar, "flows" must be ordered)
add the starting item to "visited"
add the starting item to "flows"
if the item is an unpack event
add the resourceInventoriedAs to "contained"
if the item is a modify event
add the resourceInventoriedAs to "modified"
if the item is a dropoff event
add the resourceInventoriedAs to "delivered"
call trace-depth-first-search (which will go recursively backwards through the tree)
return "flows"
trace-depth-first-search (parameters: "flows", "visited", "contained", "modified", "delivered", "savedEvent")
for the last item in "flows", get "previous" (defined below)
if that last item is an event
save the previousEvent in savedEvent
order descending the "previous" items by id or other unique element
if the savedEvent is one of those items
move it to the position where it will be processed first
for each of those items
if the item is not in "visited"
set "skip" to false
if the item is a pickup event
if the event's resourceInventoriedAs is in "delivered"
remove it from "delivered"
else
set "skip" to true
else if the item is an accept event
if the event's resourceInventoriedAs is in "modified"
remove it from "modified"
else
set "skip" to true
else if the item is a pack event
if the event's resourceInventoriedAs is in "contained"
remove it from "contained"
else
set "skip" to true
else if the item is an unpack event
add the resourceInventoriedAs to "contained"
else if the item is a modify event
add the resourceInventoriedAs to "modified"
else if the item is a dropoff event
add the resourceInventoriedAs to "delivered"
if "skip" is false
add the item to "visited"
add the item to "flows"
call trace-depth-first-search
return "flows"
resource "previous":
find all events that are process outputs and where the resource is resourceInventoriedAs
find all events that have the resource as toResourceInventoriedAs (all transfers and move)
find all raise/lower events that reference the resource as resourceInventoriedAs
return all of the above events
process "previous":
return all events that are input to the process
event "previous":
if the event is output of a process
return the process
else if the event is triggeredBy another event
return the triggered by event
else
if resourceInventoriedAs of the event exists
return the resourceInventoriedAs
Here is another diagram, in an attempt to cover more conditions for trace, minimally including covering all actions in combination with the previous diagram.
I'm not sure what will happen (and possible what should happen) with the circular dependency on polymer waste going back into the melting pot. @bhaugen
[EDIT: Note that the output of this (1 shipping bin) connects in to the first diagram, making one big flow, if we want that.]
Here is the second draft of the trace logic. I decided to do a new comment rather than replace the pseudocode above, in case we need to go back and compare easily. It still does not include the bug fix for which this issue is named. It does include logic to manage the actions that need more than the basic logic above. It is minimally desk checked, but not tested!
(note: as of now we think that fixing the bug for repeated processes of the same process spec will also handle stage and state implications - the bug fix logic will be added here later)
Note the "starting item" can be a resource or event. The event is used when wanting to start the trace earlier than the present time, for example to trace only what went into initial creation of the resource. If it is an event related to a process, it is assumed to be an output event. If you want to start the trace at the present time, use the resource to be traced.
trace (parameter: starting item)
initialize "flows", "visited", "contained", "modified", "delivered"
(list or similar, "flows" must be ordered)
add the starting item to "visited"
add the starting item to "flows"
if the item is an unpack event
add the resourceInventoriedAs to "contained"
if the item is a modify event
add the resourceInventoriedAs to "modified"
if the item is a dropoff event
add the resourceInventoriedAs to "delivered"
call trace-depth-first-search (which will go recursively backwards through the tree)
return "flows"
trace-depth-first-search (parameters: "flows", "visited", "contained", "modified", "delivered")
for the last item in "flows", get "previous" (defined below)
order the "previous" items by id or other unique element (not sure this will be needed?)
for each of those items
if the item is not in "visited"
set "skip" to false
if the item is a pickup event
if the event's resourceInventoriedAs is in "delivered"
remove it from "delivered"
else
set "skip" to true
else if the item is an accept event
if the event's resourceInventoriedAs is in "modified"
remove it from "modified"
else
set "skip" to true
else if the item is a pack event
if the event's resourceInventoriedAs is in "contained"
remove it from "contained"
else
set "skip" to true
else if the item is an unpack event
add the resourceInventoriedAs to "contained"
else if the item is a modify event
add the resourceInventoriedAs to "modified"
else if the item is a dropoff event
add the resourceInventoriedAs to "delivered"
if "skip" is false
add the item to "visited"
add the item to "flows"
call trace-depth-first-search
return "flows"
resource "previous":
find all events that are process outputs and where the resource is resourceInventoriedAs
find all events that have the resource as toResourceInventoriedAs (all transfers and move)
find all raise/lower events that reference the resource as resourceInventoriedAs
return all of the above events
process "previous":
return all events that are input to the process
event "previous":
if the event is output of a process
return the process
else if the event is triggeredBy another event
return the triggered by event
else
if resourceInventoriedAs of the event exists
return the resourceInventoriedAs
@bhaugen @srfsh please let me know if you find something wrong! Or if you want to wait for the next version, feel free of course.
[EDIT: Note the next 2 drafts are now posted below.]
First draft of pseudo-code for trace
. Still a number of questions, and it does not include the solution to the bug @bhaugen started with. But for now, it seems like we should keep this series in one place.
note the "starting item" can be a resource, process (?), or event
trace (parameter: starting item)
initialize "flows" and "visited" (list or similar, "flows" must be ordered)
add the starting item to "visited"
add the starting item to "flows"
call trace-depth-first-search, which will go recursively backwards through the tree
return "flows"
trace-depth-first-search (parameters: "flows", "visited")
for the last item in "flows", get the "previous" (see below for "previous")
order the "previous" items by id or other unique element (not sure this will be needed?)
for each of those items
if the item is not in "visited"
add it to "visited"
add it to "flows" ...
call trace-depth-first-search
return "flows"
resource "previous":
event that is output of a process (resourceInventoriedAs)
transfer or move event (toResourceInventoriedAs) (or maybe this is any event with a toResourceInventoriedAs?)
(questions: stage and state, add raise and lower, check other actions)
process "previous":
events that are input to the process
event "previous":
process from which it is output
resource which it affected as an input to a process (resourceInventoriedAs)
resource on a transfer or move event (resourceInventoriedAs)
triggeredBy event (vs resource?)
(questions: pack and unpack, raise and lower, other actions, stage and state)
I'm about to post the first draft of a solution, but I would love to get some early feedback before too many people take the first draft too uncritically.
Who's following this? Anybody? Maybe pop a reaction in here?
Ok, here's the first draft of a description of this use case. I've asked for more details and will add them when I get them and link to more sources.
But here are the sources I have now:
This use case is about tracking and tracing medical gowns in the Reflow project, which is about organizing circular economies in European cities. Reflow is using Valueflows to coordinate their circular resource flows. (They will probably provide more loopy use cases in the future.)
Each gown has a unique identity with a barcode that gets scanned in with every economic event.
The gowns are made from recycled cotton and then barcoded and transferred to hospitals. After a gown is worn, it gets packed into a shipping bin with other dirty gowns. The hospital transfers the bin to a laundry where the bin is unpacked. Then each gown gets washed and then packed back into the bin and transferred back to the hospital to be worn again. And packed and transferred and unpacked and washed and packed and transferred back and unpacked, again and again and again.
Every time anything happens to an individual gown, its barcode is scanned, and the economic event can be recorded. The event recording could probably be automated but I don't know if they will go that far, and it doesn't make any difference for the tracking and tracing problem.
To repeat (and repeat and repeat), both the gowns and the bin go through the same cycle of wear-pack-transfer-unpack-wash-pack-transfer-unpack. The flow path for tracking a gown must switch focus, when the gown is packed into the bin, from tracking the gown to tracking the bin, and the switch back to tracking the gown when it is unpacked. And the reverse switcheroo for tracing.
In the sources listed above, when they pack the gowns into the bins, they also add them to a lot, so tracking the lot is the same as tracking the bin. Tracking the bin, however, might be necessary if the bins also have individual identities and are reused over and over and also need to be tracked.
@lynnfoster has a lovely diagram of the whole set of flows, which she will post into the next comment below, and then I will describe the solution, and then some pseudocode-ish details (Correction: @lynnfoster is writing the pseudocode).
We ran into a new use case that cannot be tracked or traced by the current https://www.valueflo.ws/algorithms/track/ instructions.
The problem arises when the same resource goes many times through processes with the same ProcessSpecification. That situation can't be handled by the Stage, as suggested in those instructions:
When the same economic resource is both input and output of a process, sometimes a series of processes, such as for repair or quality testing or a workflow where a resource is refined through stages like writing/editing/etc, the stage must be identified, based on the kind of process the resource was last output of.
Stage will not work because it uses the ProcessSpecification, and in this case, that is the same for all of the repetitive Processes.
People have suggested timestamps.
The problem with using timestamps for sequencing resource flows is that we are working in distributed systems, and it can be difficult or impossible to sequence using timestamps in distributed systems. The standard way to do it is called causal ordering, which is the way the VF track and trace logic is described in the instructions.
Here are some references where you can learn a lot more, or you can just believe temporarily that the problem with timestamps is real.
But when the same resource goes through the same types of processes over and over, the methods we have used for causal ordering don't work.
First I'll describe this fascinating use case. Then I'll describe a possible solution, and in separate follow-on comments, step through some pseudocode, first to demonstrate the problem, and then to demonstrate the solution.
Another couple of attempts to clarify:
"Backwards" and "forwards" mean in terms of resource flows, not time. Tracing "backwards" from a Resource means "where did this resource come from", "how was it created", "what other resources went into it", etc. Those previous flows did also occur previously in time, but time is not the critical relationship.
Tracking a resource "forwards" means "where did this resource go?" and "what happened to it?" Did it get transported to some other place? Did it become an input to another process? Was it transferred to another agent? What did they do with it? Etc. And as with tracing, those flows occurred later in time, but again, time is the not the critical relationship. The relationships of the flows, through their input-process-output patterns, is what matters.