--- title: "Clean up data" canonical: "https://help.vicon.com/space/Shogun18/13664555/Clean%20up%20data" format: markdown --- If your visual assessment and Vicon Shogun Post's diagnostic tools have revealed issues with your recorded data (see [Check data quality](https://vicon-help.atlassian.net/wiki/spaces/Shogun18/pages/13664740)), you can use its cleanup tools to correct mislabels and fill any gaps. > ⚠️ **Important** > ⚠️ Before filling gaps, you must find and fix any labeling swaps or errors in your data. Gap-filling in Shogun relies on the data before and after the gaps to predict marker location, so any labeling errors will cause gap-filling to fail. The following topics provide an introduction to fixing data issues. > Macro (scroll-only) > > > Macro (legacy-content) > Macro (anchor) See also the related Vicon videos: [5 - Shogun Post – Labeling Data](https://www.youtube.com/watch?v=BBCqKjJ4ASc) and [6 - Shogun Post - Marker Editing](https://www.youtube.com/watch?v=ZoYldRNvWuc). > Macro (scroll-ignore) > > > Macro (widget) > Macro (scroll-ignore) > > > Macro (widget) Note that each time you finish cleaning up a range, you must check its solve (see [Solve during cleanup](#SolveDuringCleanup)). ## Find and fix bad data To help you identify and fix bad data, you can use the **Marker Editing** panel. **To find and remove bad data:** 1. To open the **Marker Editing** panel, in the **Processing** menu, click **Marker Editing** (or click **Marker Editing** on the **Processing** tab of the ribbon). 2. In the **Remove Bad Data** section, go to the **Find Bad Data** button and notice the options: - **Threshold** Allowable deviation - **Cut-Off** Data filter. Decreasing this value filters the data more heavily. - **Sensitivity** Amplifies the effect of the **Cut-Off** filter. 3. Click **Find Bad Data**. 4. Experiment with these values to find out what works best for your data. 5. When you have identified the bad data, remove it by clicking the appropriate **Cut** button, and apply a fill (see [Gap-filling options](#GapFillOptions)). ## Find and fix noise Noise can be easily identified in a **Graph** view: ![image](media://b5b25ade-4d8f-4acb-a20b-f33de59d185f) It also produces flickering as you play through a take in the **3D Scene** view. You can fix it with the filtering options in the **Marker Editing** panel. **To fix noisy data:** 1. In a **Graph** view, ALT+drag to select it. 2. At the bottom of the **Marker Editing** panel, expand the **Filter** section, and select whether filtering applies to **Ranges**, as above, or to **Selected Keys**. 3. If you're not sure how much filtering to apply to your data, keep the default settings (**Cut Off**: 0.3 and **Threshold**: 15) and click **Apply**. You can reapply this as many times as required. 4. In the **Graph** view and **3D Scene** view, check that the trajectory is now smooth over the selected range. :check_mark:** Tip: **To display a tooltip for any of the controls, hover the mouse pointer over the relevant control. > Macro (anchor) ### Ensure smooth start and end to filtered trajectories When you apply filtering, the **Smooth In/Out** option in the **Marker Editing** panel provides smooth blending at the start and end of the filtered section of a trajectory curve. The following example shows a Graph view of a trajectory curve that requires smoothing. ![image](media://81c0a307-0774-49fa-9e76-605b3feef351) With **Smooth In/Out** cleared, when filtering is applied, the curve is made smoother, but the start and end does not take into account what is either side of the smoothed section, resulting in a straight line, with an unwanted angle at the end. ![image](media://b5c322a3-9ec0-4403-8faa-052b58c926d1) > Macro (inline-media-image) For details, see the following steps or watch [Vicon Shogun 1.3 Post Tutorial - Filter in/out Smoothing](https://youtu.be/0csCqSmKe9E) on YouTube. > Macro (scroll-ignore) > > > Macro (widget) > Macro (scroll-pagebreak) To give a smoother start and end to the filtered section of a trajectory: 1. In the **Marker Editing** panel, expand the **Filter** section and select **Smooth In/Out**. 2. If you want to change how much of the trajectory is affected on either side of the gap, edit the value of the adjacent field. The default is 0.2 (20%). Smoothing works by blending between the original and filtered result by progressively more or less, across the smoothing range. The filtered section is smoothed at the start and end.> Macro (inline-media-image) > Macro (anchor) ## Use the Labeling panel You can fix labeling issues using the tools in the **Labeling** panel. The following steps introduce the main components of the labeling panel. To go straight to examples of how to use it to fix labeling issues, see [Fix common labeling issues](#FixCommonIssues). **To use the Labeling panel:** 1. On the **Processing** tab on ribbon, click **Labeling**. 2. In the toolbar at the top of the Shogun Post window, ensure that the required subject is selected in the **Current Subject** list. :check_mark: **Tip: ** By default, the selection in the **Current Subject** list at the top (middle) of the Shogun Post window determines whichsubject to label. If **All** is selected in the **Current Subject** list, the labeler uses the last subject it was set to. If you want to select a subject different from that specified in the **Current Subject** list, in the **Labeling** panel, clear the **Use Current Subject** box and select the required subject from the **Subject** list at the left of the check box. On the left of the **Labeling** panel, a list of labels for the selected subject is displayed. The color variations indicate marker issues: - **Yellow**: Mislabels or missing labels (the depth of the color indicates the severity of the issue, eg, more or fewer gaps) - **Red**: Labels for this marker are missing from the current frame 3. To display a 3D representation of your labels for the current subject, click the** 3D** button at the top of the **Labeling** panel. The 3D view helps you to quickly identify where the markers should be. You can drag and drop labels from the 3D Labeling view to the **3D Scene** view pane (and vice versa). You can use the usual mouse actions (click and drag, right-click and drag, left- and right-click and drag) in the 3D view, in the same way as in a **3D Scene** view. 4. At the top of the **Labeling** panel, in the **Manual Labeling Options** section, select the **Mode** option (**Select** or **Label**), which affects the way in which you select and label markers. - **Select** Click a marker on your subject in a **3D Scene** view and then click a label name in the list in the **Labeling** panel. - **Label** Click name of the label in the marker list and then click the required marker on the subject in the **3D Scene** view. :check_mark:** Tip: **To quickly switch between labeling and select modes when labeling a subject, you can use the default hot key (L) that duplicates the functionality of the **Label** and **Select** buttons at the top of the **Labeling** panel. 5. In the **Direction** line, choose to label forward > Macro (inline-media-image) (through time) or backward > Macro (inline-media-image) . You can also select both options (ie, label both forward and backward), but to help you avoid confusion, at least initially, choose either backward or forward and use only that option. 6. From the **Type** options, select the way in which labeling will be applied: - **Whole** Labels entire trajectory. - **Fragment** Labels the trajectory that intercepts the current frame. - **Cliff** Labels the current frame and continues until a specified value is encountered, which stops the labeling (see the text below in the **Labeling** panel, for example, the default is to skip gaps smaller than 5 frames and stop labeling at cliffs that are larger than 50 mm). - **Ranges** Lets you select an area on your graph or timeline and label only the selected time range. In the **Manual Labeling Tools** section in the middle of the **Labeling** panel, you can correct swaps, unlabel information, and unlabel markers. In the **Semi-Automated Labelers** section, you can access the Velocity Labeler, which is normally used after automated labeling, on a partially labeled take. You can use it to correct labeling where the path of a single marker consists of multiple trajectories that are consecutive in time with a small gap in between where the marker is unlabeled over part of its trajectory. The **Velocity Label** option is useful when a marker has been labeled for a range of time, and then becomes unlabeled, yet going forward or backward in time there are multiple trajectories that do not have a large gap between them and are all the same marker. It is best suited for cases when unlabeled trajectories are not many frames away from the labeled marker and the velocity of the marker around the end of the labeled marker and the start of the unlabeled trajectory is fairly constant. > Macro (anchor) ## Fix common labeling issues The following basic procedures are just a few of the ways in which you can use Shogun Post to clean up data. See also: > Macro (inline-media-image) [Vicon Shogun 1.3 Post Tutorial - Gap List & Auto Rigid Fill](https://youtu.be/U3jmKTqykOE) on YouTube. > Macro (scroll-pagebreak) > Macro (scroll-ignore) > > > Macro (widget) ### Identify issues 1. To help you identify issues, in the **3D Scene** view, click **View Filters**, and ensure that in the **Scene** section, **Missing Markers** is selected. Also ensure that in the **Subjects** section, **Label** is selected and that in the **L**(abeling) column of the views matrix, **Constraints** is selected. 2. In **3D Scene** view, scrub through the whole take (drag the current time indicator along the timeline, or for finer control, press the A or S key), noting times when markers are likely to be occluded (sitting, crouching, subjects interacting, etc). 3. In the marker list in the **Labeling** panel, note any missing (red) markers and watch for ‘popping’ or moving in the **3D Scene** view. Look for any swapped markers (for an example, see the Vicon video [5 - Shogun Post – Labeling Data](https://youtu.be/BBCqKjJ4ASc), which shows an example of a swap of knee and heel markers). Missing markers are clearly visible, highlighted in red. However, after you’ve identified the erroneous markers, it may help with labeling to clear **Missing Markers** and **Labeling Constraints**. ### Correct a swap 1. Ensure the **Labeling** panel is displayed. 2. Split the **3D Scene** view (at the top right of the Shogun Post window, click the vertical split button > Macro (inline-media-image) ) and in one of the panes, change to a **Graph** view, so that you can now see a ** 3D Scene ** view, a ** Graph ** view and the ** Labeling ** panel. > Macro (inline-media-image) 3. In the **Labeling** panel, ensure **Select** mode, Forward direction (**>**), and **Cliff** are selected. 4. In the **3D Scene** view, select a marker that is incorrectly labeled and in the **Graph** view, zoom in (right-click and drag) and go to the start of the where the swap happens (this should be identifiable on the graph by a sharp change). 5. From the position of the marker in the **3D Scene** view, decide which is the correct marker and click it in the 3D view. In the marker list, its name is displayed in heavier text, and at the top left of the **3D Scene** view, the **Selection** text displays the name of the selected marker. 6. Click the wrongly labeled marker in the **3D Scene** view and then click the correct marker name in the **Labeling** panel list. The marker is now correctly labeled. To check, scrub back and forward in the **3D Scene** view and check that the marker is now behaving correctly. However, you now have a previously incorrectly labeled marker that is unlabeled from this point forward. ![image](media://449c20e1-7518-4fbe-9cf7-78fe6150e5da) > Macro (scroll-pagebreak) ### Label an unlabeled marker 1. In the **3D Scene** view, click on the unlabeled marker to select it. Scrub back and forward to identify where it is correctly labeled. This is the point at which it is correctly colored in the **3D Scene** view and the **Labeling** panel marker list. You can also observe this on the **Graph**. 2. To label a marker using the tools in the **Labeling** panel, do one of the following, depending on your chosen **Mode**: If you prefer to label using **Label** mode: 1. In the **Manual Labeling Options** section, in the **Mode** line, click **Label**. 2. In the marker list in the **Labeling** panel, click the name of the marker and then in the **3D Scene** view, click the marker that is currently unlabeled. Or, if you prefer to label by dragging, using **Select** mode: 1. In the **Manual Labeling Options** section, in the **Mode** line, ensure **Select** mode is selected. 2. In the **Labeling** panel, click to select the required marker in the **3D** view. 3. SHIFT+CTRL then click+drag from the selected marker in the **Labeling** panel to the unlabeled marker in the **3D Scene** view. 3. As before, to check the labeling, scrub back and forward in the **3D Scene** view and check that the marker is now behaving correctly. After you have corrected any swaps, you may want to use Shogun Post's diagnostic tools to help to identify gaps (see [Check data quality](https://vicon-help.atlassian.net/wiki/spaces/Shogun18/pages/13664740)). ### View and select gaps You can view all the gaps in your scene in the **Marker Editing** panel, in the **Fill Gaps** section, by looking at the **Gap List**. (If you can't see the list, at the top of the **Marker Editing** panel, click the **Gap List** button.) ![image](media://05600956-e895-4cf9-b548-f006e09dbe1b) To make it easy to find the longest gaps, click the **Length** column heading to rearrange the list, with the longest gap at the top. ![image](media://0c35ba56-a756-415f-a422-890f83f0891a) When you select a gap in the list, it is automatically displayed in the Graph view, enabling you to edit it. ![image](media://4078c1b8-2084-419e-bb17-799bf3e2e5e0) > Macro (scroll-pagebreak) You can then decide the best approach for filling the gaps: - As a first step, try automatically filling as many gaps as possible (see [Auto-fill with intelligent rigid fill](#AutoFillRigid)). - If your scene still contains gaps after using the automatic fill, or if auto-fill is unsuitable, try manually filling the gaps (see [Manually fill gaps](#ManualGapFill)). > ✅ **Tip** > ✅ To undo unwanted gap-filling, click Undo on the quick access toolbar > Macro (inline-media-image) > ✅ > ✅ . > Macro (anchor) ### Auto-fill with intelligent rigid fill Shogun Post enables you to automatically fill gaps using a rigid fill operation. This looks at all the markers in your scene and then compares them against the marker you are trying to fill. It then uses a combination of similarly moving markers to fill the gap. Finally, it checks the fill to make sure it looks correct and if not, it chooses another set of markers. ![image](media://03607211-28e8-42f5-a7ec-f14bc3f83b20) This process is also available via scripting, using the [selectMarkersForRigidFill](https://vicon-help.atlassian.net/wiki/spaces/Shogun18/pages/13668029) command to select the markers and the [autoFillGaps](https://vicon-help.atlassian.net/wiki/spaces/Shogun18/pages/13668884) command to fill them. For information on these commands, see *HSL scripting with Vicon Shogun*. > Macro (scroll-pagebreak) **To auto-fill a selected marker:** The following procedure for rigid gap-filling is semi-automated, in that you choose a marker for the fill: 1. In the **Marker Editing** panel, expand the **Fill Gaps** section and in the **Fill Rigid** area, select the **Auto-select markers for fill** check box. To change the default options for auto-selection, click the **Show Options** button next to the check box. The options are: - **Max Deviation**: Specifies in mm how rigid the set of markers used to fill must be, as the maximum deviation in distance between all pairs in the set. For example, a value of 60 enables the markers to flex in rigidity by 6 cm. - **Max Distance**: Specifies in mm how far away the candidate can be from the selected marker. The default of900 is just under 3 feet (1 m), because a greater distance is likely to be another subject or body part, which is unlikely to move rigidly during the gap. - **Percent Time Rigid**: Specifies the percentage of time over the adjacent fragment ranges that the selected marker must be rigid with the chosen set. The default of 0.75 means 75% of the time. 2. Select a marker with a gap. 3. Ensure the current time is in the gap. 4. Click **Fill Rigid**. Gaps are filled using the intelligent rigid fill. **To auto-fill selected markers or all markers:** In addition to the semi-automated procedure described above, you can also automatically fill all gaps on either all markers, or only those currently selected. To do this: 1. Depending on which markers you want to fill: - **Only selected markers**. Ensure you have selected the required markers. - **All markers**. Ensure no markers are selected. 2. In the **Marker Editing** panel, expand the Fill Gaps section and in the **Fill Rigid** area, select the **Current Frame**, **Selected Ranges** or **All Time** option. 3. Select **Auto-select markers for fill** and ensure the options are as required (see above). 4. Click **Fill Rigid**. All gaps on the selected markers, or on all markers if no markers are selected, are filled over the play range, or selected ranges. Note that in the case of all markers, only labeled markers are filled, though unlabeled markers can be auto-selected for filling the labeled marker. > ℹ️ **Note** > ℹ️ After running automated gap-filling, if Shogun did not find a set of rigid markers based on the settings, you may find that not all gaps are filled. You can either fill the remaining gaps manually, or you can try changing the settings, specifically by increasing **Max Deviation** from the default. You can run automated gap-filling multiple times to achieve the required results, in particular when: - After first run, it either made an unwanted fill or wasn't able to fill a gap due to a mislabel. Fix the mislabel and then run it again to fill the remaining gaps. - After first run, it either made an unwanted fill or wasn't able to fill a gap due to the gap being too complex. Fill the gap manually, then run automated gap-filling again to fill any remaining gaps. These can now can be filled due to more data being available. - After first run, some gaps remain. Loosen the rigidity settings (see Note above) and run it again. > Macro (scroll-pagebreak) > Macro (anchor) ### Manually fill gaps You fill gaps using the **Marker Editing** panel, normally in conjunction with the **Data Health** view and/or a **Graph** view. The **Marker Editing** panel contains all the tools necessary to fill gaps, alter trajectory keys and filter your data. For example, you might first notice a gap from the display on the time bar Issues map, then note which marker is affected in the **3D Scene** view: ![image](media://4751f2e3-fef0-4ab8-9158-aa9080f48eaf) 1. To quickly gap-fill the selected marker, on the time bar double-click to move the current time indicator to the relevant frame. The affected marker is automatically highlighted in the **3D Scene** view (you can right-click and drag to zoom in further to check the marker). 2. To open a **Data Health** view and a **Graph** view, as well as a **3D Scene** view, split the screen by clicking the Three Views Split Left button > Macro (inline-media-image) at the top of the Shogun Post window and change the views in the new panes. 3. If the **Marker Editing** panel is not already open, on the **Processing** tab on the ribbon, click **Marker Editing**. 4. In the**Data Health** view, find the relevant marker's line, which shows a gray rectangle that represents the gap. 5. In the **Data Health** view, double-click to select the range of the gap. 6. In the **Marker Editing** panel, expand the **Fill Gaps** section. In the **Fill Using Interpolation** area, select **Selected Ranges** (to fill only the current selection), and then click **Fill Using Interpolation**. This looks at the keys on either side of the gap and fills it by interpolating between them. It works well for small gaps. 7. In the **3D Scene** view and **Graph** view, check that your fill has given the required result. 8. If the fill does not give an appropriate result, you can click Undo on the quick access toolbar > Macro (inline-media-image) , and try other gap-filling options in the **Marker Editing** panel (see [Options for gap filling](#GapFillOptions)). > Macro (scroll-pagebreak) > Macro (anchor) #### Options for manual gap-filling The appropriate option to use depends on the type of gap that you want to fill. > Macro (scroll-only) > > > Macro (toc) ##### Short gaps 1. Manually select a short gap by double-clicking in a **Graph** view or **Data Health** view, and in the **Marker Editing** panel, expand the **Fill Gaps** section and click **Fill Using Interpolation**. This looks at data before and after the gap and uses a spline fill to interpolate between them. 2. Check that the result looks reasonable in a **3D Scene** view and **Graph** view. ##### Sequence of short gaps 1. For a sequence of small gaps with a smooth line running through the gap in the **Graph** view, ALT+drag to select the affected data. 2. In the **Fill Using Interpolation** area, select **Selected Ranges**, and then click **Fill Using Interpolation**. > Macro (scroll-pagebreak) ##### Non-linear motion For non-linear motion, for example, where rotational movement is involved, as interpolation does not account for this kind of motion, a different approach is needed. If a missing marker is part of set of markers that are rigid (ie, remain in the same relation to each other, for example, a pelvis), you can use three or more markers in the rigid object (three in addition to the marker that has the gap is recommended) to help fill in the data for the missing marker. 1. To select the range of the gap, double-click in a **Graph** view or **Data Health** view, then select (CTRL+click) the three other markers in the rigid object. 2. In the **Marker Editing** panel, expand the **Fill Gaps** section. 3. In the **Fill Rigid** area, select **Selected Ranges** and click **Fill Rigid**. The pattern of the present markers is copied onto the missing marker. 4. As usual, check in the **Graph** view and in a **3D Scene** view that the fill looks sensible. Other rigid objects you can use in default marker set include hands, forearm, upper arm, shoulders, thorax, and head. > Macro (scroll-pagebreak) ##### Non-linear motion where rigid fill impossible In cases where the motion is non-linear, but a rigid fill is not possible due to all the related markers being absent, you can use the **Fill Using Constraints** option. This uses the labeling or solving skeleton to produce the fill. It requires a skeleton be present, the skeleton to be constrained by markers, and enough markers to have data that the solve is fairly good. The marker being filled must be constrained to the skeleton because it is the constraint offset that is used to determine where the marker should be during the fill. Before using this type of fill, check that the bones and missing markers (by default displayed in red in the **3D Scene** view), are in sensible locations. ## Processing during cleanup Note that the previous steps for fixing data issues assume that your existing .*mcp* data is largely of acceptable quality. However, in some circumstances (for example when you are trying to produce better quality data), rather than persisting in trying to fix a problematic .*mcp* file, you may need to clear the scene and start from the .*x2d* file, or unlabel all the data first. In this case, do not just open the **Processing** panel and click **Reconstruct** or **Label** or run other **Combined Processing** operations as this will not fix problems with the underlying data. ### Clear existing data To remove all unlabeled data and clear existing labels and solves, at the top of the **Processing** panel, select the **Reset Scene** check box. You can then run **Reconstruct**, etc. Note that you can't undo the reset. To remove existing labels from the current file, on the** Labeling** tab of the **Processing** panel, select **Clear Existing Labels** and then run **Labeling** as required. To remove occlusion fixing from your data, in the **Marker Editing** panel, expand the **Restore** section and select the required option. > Macro (anchor) ### Solve during cleanup During the cleanup operation, each time you finish cleaning up a range, check its solve. To do this: 1. Ensure the range you cleaned up is selected. 2. On the **Processing** tab on the ribbon, click the arrow on the **Solve Solving** button. 3. Click **Solve Solving Ranges**. The selected range is solved, enabling you to quickly check that your cleanup has been successful, without having to solve the whole take. 4. Proceed to clean up the next range that contains mislabels, gaps or noisy markers, etc. After you have completed all the cleanup required, finish by solving the entire take. For more information see [Solve the data](https://vicon-help.atlassian.net/wiki/spaces/Shogun18/pages/13664542).