Domain scheduling

Keeping a close watch on your website and staying updated on all changes occurring on every page is really important. Having such a deep insight allows you to monitor your website closely and indicate wrong changes in your website sooner. Furthermore, it increases your SEO and ranking. This is when you need an automated way to monitor your website’s pages periodically and analyze its content to have a deeper insight of what happens in your website. Furthermore, this features helps to gather data to enrich different modules of URLsLab plugin.

What is Domain Scheduling?

Domain Scheduling is a section in URLsLab Plugin that shows you what domains have you scheduled. The schedules impact the synchronization between URLsLab Plugin and URLsLab Service. in Schedules section of Domain Scheduling you will see all your schedules. you can either delete them or add a new schedule based on your needs.

Adding New Schedule

You can add a new schedule both in URLsLab Dashboard and URLsLab Plugin dashboard. to add a new schedule head to Domain Scheduling and click on Add Schedule. In the popup you can specify the behaviors and configuration, in which the URLsLab Service will crawl your website.

In the following we will explore each option and give a detailed description, that how will they affect your website’s scanning.

• Domain URL: This is the Home page of your URL that URLsLab Service will start to crawl. Here, only the homepage URL should be specified, (eg. https://www.urlslab.com)
• Analyze Text: With each crawl, URLsLab can fetch the content of your webpages, index them, generate summary of each page, which can enhance the data that is used in webpage. below, we will discuss, the different modules that this configuration affects:
  • Summary Generation: When Analyze Text is set in the configuration, URLsLab Service generates a summary of each page of your domain. The same summary can be used in Related Articles (For including the content of linked webpage), Meta Tags Manager (For replacing the og description meta tags with page summary), Link Building (For replacing the destination page summary with the link title)
  • Finding Related Articles: This setting will also make sure that all your page’s content is indexed, which make sit possible to find the best matching related webpage to any of your webpages. this setting enables us to index your content more efficiently to find the best related pages.

• Process found links: When this setting is set, URLsLab Service, will crawl the internal links that are found in your webpages, and uses the links in your pages to discover new URLs. this is an alternative way of discovering urls in addition to sitemap. We recommend to set this setting to make sure all your webpages has been crawled.
• Domain Sitemaps: When set, URLsLab Service, will use the sitemap specified in Robots.txt to discover new URLs and crawl them. We recommend to set this setting to make sure all your webpages has been crawled.
• Screenshots: When set, URLsLab Service, will take screenshot of the scheduled webpages. this will enhance Related Articles (when including linked webpage’s screenshot). This feature is also important for Screenshot (Makes sure all your internal pages have been scheduled for screenshot-ting) and its the main feature of URL Monitoring(This feature enables capture of webpage history, so that you can see your webpage changes throughout time in URL Monitoring Module). We recommend to set this setting to make sure all your webpages has been crawled.

• Scan Frequency: The frequency that you want URLsLab to crawl your webpages and update indexing and screenshot data. Note that page summary will not be updated if less than 30% of the page’s content has not been changed and you will not be charged. In Addition, a new screenshot will not be taken, if no visual change has been detected in the webpage. We recommend to schedule your domain Monthly to make sure the data that URLsLab Plugin is using is up-to-date
• Scan Speed: The speed that you allow URLsLab service to crawl your domain in RPM (Request Per Minute). In order to not damage your website’s infrastructure, URLsLab service implements mechanisms to throttle scheduling of your website’s URLs. We recommend the speed of 20 RPM which should be ok for most of the web hostings. However, this should be tuned based on your website’s infrastructure capabilities.