--- title: "Clean up data" canonical: "https://help.vicon.com/space/Shogun19/13795574/Clean%20up%20data" format: markdown --- If your visual assessment and Vicon Shogun Post's diagnostic tools have revealed issues with your recorded data (see [Check data quality](https://vicon-help.atlassian.net/wiki/spaces/Shogun19/pages/13795533)), you can use its cleanup tools to correct mislabels and fill any gaps. > ⚠️ **Important** > ⚠️ Before filling gaps, you must find and fix any labeling swaps or errors in your data. Gap-filling in Shogun relies on the data before and after the gaps to predict marker location, so any labeling errors will cause gap-filling to fail. The following topics provide an introduction to fixing data issues: > Macro (scroll-only) > > > Macro (toc) > Macro (anchor) See also the related Vicon videos: [5 - Shogun Post – Labeling Data](https://youtu.be/BBCqKjJ4ASc) and [6 - Shogun Post - Marker Editing](https://youtu.be/ZoYldRNvWuc). > Macro (scroll-ignore) > > > Macro (widget) > Macro (scroll-ignore) > > > Macro (widget) Note that each time you finish cleaning up a range, you must check its solve (see [Solve during cleanup](#SolveDuringCleanup)). ## Find and fix bad data To help you identify and fix bad data, you can use the **Marker Editing** panel. **To find and remove bad data:** 1. To open the **Marker Editing** panel, in the **Processing** menu, select **Marker Editing** (or select **Marker Editing** on the **Processing** tab of the ribbon). 2. In the **Remove Bad Data** section, go to the **Find Bad Data** button and notice the options: - **Threshold** Allowable deviation - **Cut-Off** Data filter. Decreasing this value filters the data more heavily. - **Sensitivity** Amplifies the effect of the **Cut-Off** filter. 3. Select **Find Bad Data**. 4. Experiment with these values to find out what works best for your data. 5. When you have identified the bad data, remove it by selecting the appropriate **Cut** button, and apply a fill (see [Gap-filling options](https://vicon-help.atlassian.net/wiki/spaces/Shogun19/pages/13795577/Fix+common+labeling+issues#GapFillOptions)). ## Find and fix noise Noise can be easily identified in a **Graph** view: ![image](media://54a60681-7b2c-45f0-aa46-1586925703aa) It also produces flickering as you play through a take in the **3D Scene** view. You can fix it with the filtering options in the **Marker Editing** panel. **To fix noisy data:** 1. In a **Graph** view, ALT+drag to select it. 2. At the bottom of the **Marker Editing** panel, expand the **Filter** section, and select whether filtering applies to **Ranges**, as above, or to **Selected Keys**. 3. If you're not sure how much filtering to apply to your data, keep the default settings (**Cut Off**: 0.3 and **Threshold**: 15) and select **Apply**. You can reapply this as many times as required. 4. In the **Graph** view and **3D Scene** view, check that the trajectory is now smooth over the selected range. :check_mark: **Tip**: To display a tooltip for any of the controls, hover the mouse pointer over the relevant control. > Macro (anchor) ### Ensure smooth start and end to filtered trajectories When you apply filtering, the **Smooth In/Out** option in the **Marker Editing** panel provides smooth blending at the start and end of the filtered section of a trajectory curve. The following example shows a Graph view of a trajectory curve that requires smoothing. ![image](media://93224e4a-d4c3-4fcb-a3fd-f5d010b3da92) With **Smooth In/Out** cleared, when filtering is applied, the curve is made smoother, but the start and end does not take into account what is on either side of the smoothed section, resulting in a straight line, with an unwanted angle at the end. ![image](media://10c9a40a-618e-47be-8b5e-f699cee67ff8) > Macro (inline-media-image) For details, see the following steps or watch [Vicon Shogun 1.3 Post Tutorial - Filter in/out Smoothing](https://youtu.be/0csCqSmKe9E) on YouTube. > Macro (scroll-ignore) > > > Macro (widget) > Macro (scroll-pagebreak) To give a smoother start and end to the filtered section of a trajectory: 1. In the **Marker Editing** panel, expand the **Filter** section and select **Smooth In/Out**. 2. If you want to change how much of the trajectory is affected on either side of the gap, edit the value of the adjacent field. The default is 0.2 (20%). Smoothing works by blending between the original and filtered result by progressively more or less, across the smoothing range. The filtered section is smoothed at the start and end. > Macro (anchor) ## Use the Labeling panel You can fix labeling issues using the tools in the **Labeling** panel. The following steps introduce the main components of the labeling panel. To go straight to examples of how to use it to fix labeling issues, see [Fix common labeling issues](https://vicon-help.atlassian.net/wiki/spaces/Shogun19/pages/13795577). **To use the Labeling panel:** 1. On the **Processing** tab on ribbon, select **Labeling**. 2. In the toolbar at the top of the Shogun Post window, ensure that the required subject is selected in the **Current Subject** list. :check_mark: **Tip**: By default, the selection in the **Current Subject** list at the top (middle) of the Shogun Post window determines which subject to label. If **All** is selected in the **Current Subject** list, the labeler uses the last subject it was set to. If you want to select a subject different from that specified in the **Current Subject** list, in the **Labeling** panel, clear the **Use Current Subject** box and select the required subject from the **Subject** list at the left of the check box. On the left of the **Labeling** panel, a list of labels for the selected subject is displayed. The color variations indicate marker issues: - **Yellow**: Mislabels or missing labels (the depth of the color indicates the severity of the issue, eg, more or fewer gaps) - **Red**: Labels for this marker are missing from the current frame 3. To display a 3D representation of your labels for the current subject, click the** 3D** button at the top of the **Labeling** panel. The 3D view helps you to quickly identify where the markers should be. You can drag and drop labels from the 3D Labeling view to the **3D Scene** view pane (and vice versa). You can use the usual mouse actions (click and drag, right-click and drag, left- and right-click and drag) in the 3D view, in the same way as in a **3D Scene** view. 4. At the top of the **Labeling** panel, in the **Manual Labeling Options** section, select the **Mode** option (**Select** or **Label**), which affects the way in which you select and label markers. - **Select**: Select a marker on your subject in a **3D Scene** view and then select a label name in the list in the **Labeling** panel. - **Label**: Select the name of the label in the marker list and then select the required marker on the subject in the **3D Scene** view. :check_mark: **Tip**: To quickly switch between labeling and select modes when labeling a subject, you can use the default hot key (L) that duplicates the functionality of the **Label** and **Select** buttons at the top of the **Labeling** panel. 5. In the **Direction** line, choose to label forward > Macro (inline-media-image) (through time) or backward > Macro (inline-media-image) . You can also select both options (ie, label both forward and backward), but to help you avoid confusion, at least initially, choose either backward or forward and use only that option. 6. From the **Type** options, select the way in which labeling will be applied: - **Whole** Labels entire trajectory. - **Fragment** Labels the trajectory that intercepts the current frame. - **Cliff** Labels the current frame and continues until a specified value is encountered, which stops the labeling (see the text below in the**Labeling** panel, for example, the default is to skip gaps smaller than 5 frames and stop labeling at cliffs that are larger than 50 mm). - **Ranges** Lets you select an area on your graph or timeline and label only the selected time range. In the **Manual Labeling Tools** section in the middle of the **Labeling** panel, you can correct swaps, unlabel information, and unlabel markers. In the **Semi-Automated Labelers** section, you can access the Velocity Labeler, which is normally used after automated labeling, on a partially labeled take. You can use it to correct labeling where the path of a single marker consists of multiple trajectories that are consecutive in time with a small gap in between where the marker is unlabeled over part of its trajectory. The **Velocity Label** option is useful when a marker has been labeled for a range of time, and then becomes unlabeled, yet going forward or backward in time there are multiple trajectories that do not have a large gap between them and are all the same marker. It is best suited for cases when unlabeled trajectories are not many frames away from the labeled marker and the velocity of the marker around the end of the labeled marker and the start of the unlabeled trajectory is fairly constant. ## Processing during cleanup Note that the previous steps for fixing data issues assume that your existing .*mcp* data is largely of acceptable quality. However, in some circumstances (for example when you are trying to produce better quality data), rather than persisting in trying to fix a problematic .*mcp* file, you may need to clear the scene and start from the .*x2d* file, or unlabel all the data first. In this case, do not just open the **Processing** panel and click **Reconstruct** or **Label** or run other **Combined Processing** operations as this will not fix problems with the underlying data. ### Clear existing data To remove all unlabeled data and clear existing labels and solves, at the top of the **Processing** panel, select the **Reset Scene** check box. You can then run **Reconstruct**, etc. Note that you can't undo the reset. To remove existing labels from the current file, on the** Labeling** tab of the **Processing** panel, select **Clear Existing Labels** and then run **Labeling** as required. To remove occlusion fixing from your data, in the **Marker Editing** panel, expand the **Restore** section and select the required option. > Macro (anchor) ### Solve during cleanup During the cleanup operation, each time you finish cleaning up a range, check its solve. To do this: 1. Ensure the range you cleaned up is selected. 2. On the **Processing** tab on the ribbon, select the arrow on the **Solve Solving** button. 3. Select **Solve Solving Ranges**. The selected range is solved, enabling you to quickly check that your cleanup has been successful, without having to solve the whole take. 4. Proceed to clean up the next range that contains mislabels, gaps or noisy markers, etc. After you have completed all the cleanup required, finish by solving the entire take. For more information see [Solve the data](https://vicon-help.atlassian.net/wiki/spaces/Shogun19/pages/13795580).